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ABSTRACT 



This report presents the results of an evaluation of the Cleff Job 
Hatching System (COMS) conducted by UUrasystems, Inc. for the United 
States Department of Labor. The Cleff Job Matching System provides 
a means by which jcbs and job applicants can be matched at the semi 
and low skilled levels In both white and blue collar jobs. In the 
words of the system's principal developer, Or. Samuel H. Cleff, the 
CJMS was developed based on the following basic principle: 

"In looking fov work (as elsewhere in life)^ people seek 
out those activities in which they feel they are more 
likely to be successful. Conversely^ tfiey avoid those 
activities in which they feel less likely to be success- 
ful, Occttpationally well-adjusted people — those who like 
what they do and believe ihey are doir^ it well — are 
significantly more likely to do a better sob for their 
ejnptoyerst and to stay on their jobs longer^ than people 
who a2>e ocaupationally maladjusted— i^ose ufto do not 
like whjct they do and believe they are not doing well," 

The COMS operates by obtaining numerical profiles of job seekers and 
jobs in terms of 16 "Dimensions of Work." These 16 Dimensions of 
Work are grouped into three behavioral categories: Thing-oriented 
activites {8 dimensions). People-oriented activites {4 dimensions) 
and Idea-oriented activities {4 dimensions). Job seeker profiles are 
obtained from two Self Interview Checklists which are self completed. 
One Self Interview Checklist is used to obtain the profile which in- 
dicates those activities which they^ the job seeker, like best and 
dislike most {i.e., what motivates them positively and negatively); 
the other Indicates those activites which they have done most and done 
least. These two profiles are called the Preference and Experience 
profiles respectively. A combined job seeker profile is then ob- 
tained by averaging each of the individual dimension scores from the 
Preference and Experience profiles. 

A profile Is also obtained which describes th^ specific requirements 
of each job in terms of the same 16 Dimensions of Work used In the 
job seeker profile. The job seeker and job profiles are compared by 
correlation for similarity and by a difference statistic for major 
differences. The system operates by calculating the correlation and 
difference indices between a given job seeker profile and a series 
of job profiles or vice versa. In tKIs way job seeker-job matches 
can be rank-ordered in terms of the closeness of the match between 
job seeker and job profiles. In addition the two job seeker profiles 
{Preference and Experience) are compared using correlation and a dif- 
ference to obtain measures of the match between an Individual's 
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The indices provided by the CJKS are significantly related 
to job retention. In particular, the results show that the 
Job .'latch Index {the correlation between the combined job 
seeker profile and the job profile) is significantly related 
to the probability that the placed individual will remain em- 
ployed through the initial period of employment, i.e., over 
the first 120 days. The results also show that it is the 
Client Adjuslanent Index {the correlation between the job 
seekers Preference and Experience profiles) which relates to 
long-term retention, i.e., employment lasting over 200 days. 

Interviews were conducted by Ultrasystems with the staff of the eight 
agencies that pf.rticipated in the original Cleff jobs experiment. These 
interviews Uid to the following conclusion. 

The Cleff Job Hatohing System providen infovmation about job 
seekers that is useful in assessing individuals and providing 
vocational ootmseling. 

The validity of the Cleff Job Matching System scores as a predictor 
of retention potential of job/job seeker matches and the responses 
of personnel interviewed regarding the system's usefulness lead 
Ultrasystems to the following conclusion: 

It is Htmsystem ' opinion that the Cleff Job Matching 
System should be aonsidered for use in ike labor exdkange 
activities of the U.S. Employment Service, CETA prime 
sponsors, and other manpauer program. 

In addition, it is Ultrasystems opinion that there are facets of the 
CJMS that warrant improvement. These are discussed in Volume I of the 
final report. 

The final report has been dividec* iuto two volumes. Volume I contains 
a brief introduction followed by the findings and conclusions. In 
addition. Volume I presents a description of the Cleff Job Matching 
System and of the evaluation. 

Volume II presents the detailed analyses undertaken in this evaluation. 
The major topics covered are: 

1) The attitudes and opinions of the staff of the agencies 
in New Jersey that participated In the original OEO-Cleff 
expenment. The organization of these agencies and the 
manner in which they operated the Cleff Job Matching 
System is also discussed. 
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2) Analyses of the New Jersey agency's activity data obtained 
as part of the OEO-Cleff experiment. This also includes 

a review of the ADP-PDS documentation and analysis of the 
New Jersey-Cleff experiment, 

3) The organization and operation of the SER-C-leff experiment. 

4) Analyses of the data obtained from the SER-Cleff experiment. 
The primary analyses presented involves the detentri nation 

of the effectiveness of the CJMS in predicting the resultant 
retention of job-client matches. 
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1.0 INTRODUCTION 



The Cleff Job Matching System (CJMS) provides a means by which jobs 
and job applicants can be matched at the semi and low skilled levels 
in both white and blue collar jobs. In the words of the system's 
principal developer, Dr. Samuel H. Cleff> the CJMS was developed 
based on the following basic principle:' 

. "Underlying the research was a bas.lc». common sense prin- 
ciple, long part of our Industrial folklore but never 
verified scientifically and consistently ignored In prac- 
tice by both people seeking jobs and employers with jobs 
to fill. 

"The principle Is this: 

"In looking for work (as elaewhere in tife)^ peapU seek 
out those activities in which they feel they are mope 
likely to be successful. Conversely^ they avoid those 
activities in which they feel less likely to be success- 
ful. Ocaupationally well-adjusted peaple-^those who Hke 
what they do and believe they are doing it well-^are 
significantly more likely to do a better dob for their 
employers^ <md to stay on their jobs longer^ than people 
why are occt^ticmlly maladjusted-"'those who do not 
like what they do and believe they are not doing well." 

In June 1972 the Office of Economic Opportunity Office of Program 
Development/Evaluation awarded Ultrasystems a contract to evaluate 
the Cleff Job Matching System. The contract was transferred to the 
U.S. Deparlanent of Labor, Manpower Administration, Office of Policy, 
Evaluation and Research in September 1973. 

This report presents the results of the evaluation. The report has 
been divided into two parts. Vol; I contains this introductory sec- 
tion followed by the findings and conclusions, a description of 
the system and a description of the evaluation. Vol. U presents 
the technical dissertation. 

Before proceeding to the findings and conclusions Ultrasystems 
believes it Is important that the reader have a brief idea of the 
CJMS and the evaluation. The following two subsections present 
these brief descriptions. 



1 Quote taken from "Job/Han Matching In the '70's," by Dr. 

Samuel H. Cleff and Robert M. Hecht, in datamation, February 1, 
1971. This article is shown in Its entirety in Appendix A. The 
article described both the history of the system's development 
and Its characteristics. 
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1.1 



BRIEF DESCRIPTION OF THE CLEFF JOB MATCHING SYSTEM 



The Cleff Job Matching System is based on obtaining numerical scores 
from job seekers and employers on each of sixteen dimensions of work. 
These sixteen dimensions are divided into three categories: there 
are 8 dimensions for Thing-oriented activitiesi 4 for People-oriented 
activities and 4 for Idea-oriented activities. The set of 16 numerical 
dimensions obtained is referred to as a profile. The same 16 dimen- 
sions are used on both the job seeker and job profiles. 

The 16 numerical dimensions scores are obtained from job seekers via 
two Self-Interview Check Lists (SICL's). Each SICL consists of fif- 
teen pages with each page containing sixteen activity phrases; one for 
each of the sixteen dimensions of work. These activity phrases, or 
behavioral units* consist of a set of words which describe a unique 
or meaningful activity and a behavior which has some objective. Each 
phrase consists of a verb and an object. For example with regard to 
the Thing-oriented dimension Athletic one finds such phrases "Move 
Furniture," "Bale Hay or Paper»" "Do Weight Lifting." 

One checklist is used to obtain the scores that indicate what the job 
seeker likes best and dislikes most {i.e., what motivates them posi- 
tively and negatively). The second checklist indicates what the job 
seeker has done most and done least. 

For jobs the 16 Dimensions of Work are obtained from job supervisors 
via a Job Outline Check List (JOCL) and a Job Card Sort. The Job 
Outline Check List is identical in format to the job seeker Self In- 
terview Check List» but consists of ten pages. The Job Card Sort 
consists of a set of sixteen cards* one for each of the Dimensions of 
Work. The front of each card contains a definition of one of the 
dimensions of work. The back of each card contains examples of the 
activities that each dimension refers to. The cards are sorted by 
the supervisor into two sets. One of these consists of five cards 
that best describe what must be done on the job to do it right. The 
other consists of five cards which best describe those activities 
which must be avoided by a worker in the job or are least required 
by the job. The supervisor then assigns scale values to each of 
these cards. These values are combined with the scores obtained from 
the JOCL to arrive at the job profile. The JOCL describes the speci- 
fic requirements of the job in terms of the 16 Dimensions of Work. 
The card sort provides a general or global description of the job 
and serves as an internal consistency check on the job profile ob- 
tained from the JOCL. 

The two job seeker profiles are referred to as the Likes and Dis- 
likes or Preference profile and the Done and Hot Done or Experience 
profile. These two profiles are combined to obtain a Combined 
profile. The two basic job seeker profiles are also compared by 
correlation for similarity and by a difference statistic for major 
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differences in dimensional scores. These two measures are referred to 
as the Client Adjustment Index (CAI) and the Client Difference Index 
(CDI). High positive values of the Client Adjustment Index indicate 
similarity between the persons likes and dislikes and what they have 
done and not done. 

The Combined client profile is compared to the job profile using cor- 
relation for similarity and a difference statistic for major differ- 
ences in dimension scores. These two measures are referred to as the 
Job Match Index (JHI) and the Job Difference Index (JDI). Generally 
speaking a high positive Job Match Index and a low Job Difference 
Index indicate a good match of job seeker and job in terms of the 16 
dimensions of work. 

A Job Cluster Register is available based on clustering over five hun- 
dred individual job profiles obtained via the JOCL and the Job Card 
Sort. There are 19 clusters or individual job groups in the cluster 
register. Job seeker profiles are compared to the cluster profiles 
to obtain client-job matches. The resultant matches can be used to 
direct job development activities as well as for direct client-job 
matching when jobs are available that have not been profiled. 

The key part of the system in client-job matching is the comparison 
of the job seekers ombined profile with the job profile. The Job 
Match and Job Difference indices provide numerical scores that indi- 
cate the closeness of tiie match. The individual dimension scores 
are also useful in matching. The job seekers Preference and Experi- 
ence profiles also play a role in job matching independent of job 
profiles. In addition the numerical indices obtained by comparing the 
Experience and Preference profiles and tiie actual profiles tiiemselves 
tthe individual numerical dimension scores) are used in vocational 
counseling and client assessment. 

The CJMS has been developed so that jobs and job seekers are described 
in the same terms; i.e. according to the sixteen dimensions of work. 
The numerical dimension scores have positive and negative values in- 
dicating activity likes and dislikes, activities done and not done» 
and job activity requirements, not required or to be avoided. 

The checklists are self-administered pencil and paper instruments. 
The checklists can be administered orally. 

For more details regarding the Cleff Job Matching System's develop- 
ment, structure, and validation the reader is referred to Appendices 
A and B and to Section 3 of Vol. I. 
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BRIEF DESCRIPTION OF THE EVALUATION AND ANALYSIS OF 
THE CLEFF JOB MATCHING SYSTEM 



On June 30, 1972, Ultra systems. Inc. was awarded a contract for the 
evaluation and analysis of the Cleff Job-Client Hatching System by 
the Office of Econotdc Opportunity. The project was to use as the 
source of the information required by the evaluation the activities 
underway in an experimental program funded by OED and being conducted 
in conjunction with eight manpower agencies/offices in New Jersey 
(three community agencies and five New Jersey Vocational Rehabilita- 
tion Offices). The experimental program was run by ADP-Personnel 
Data Systems, Inc.. (ADP-PDS), under contract to OEO. 

The experiment had been designed to obtain comparative information 
between two groups of clients using the participating agencies for 
job and training placement services. One group of clients would be 
given the Self-Interview Check Lists of the Cleff System. The other 
would not be given thechecklists, but would be given services in the 
normal agency manner. The evaluation was to determine if the use of 
the CJHS to direct referral activities resulted in a significantly 
improved "level" of referral/placement "success" as evidenced by: 

1. A higher rate of successful referral outcomes 
(greater percentage of referrals resulting in 
hires or training program enrollments). 

2. A significant improvement in post-placement results 
as evidenced by a higher level of retention, worker 
job satisfaction and employer rating of worker job 
performance. 

The evaluation involved determining not only comparative performance 
between the two client groups on the above measures but, in the opinion 
of Ultrasystems, more importantly in terms of what the CJHS itself 
predicted, through its scoring system, to be the expected results. 
That is, the scores relating the client and the job or training pro- 
gram they were referred to and placed in were to be tested versus the 
results in terms of the above measures. The data relative to reten- 
tion, on-job performance and worker satisfaction were to be obtained 
by Ultrasystems through interviews with clients from both groups who 
were successfully placed and from their respective employers. The 
data needed to compare referral/placement success and needed to as- 
certain client/job CJHS scores was to be furnished Ultrasystems (in 
machine-readable form) by the compar\y contracted to run the experiment. 

Shortly after award of the contract, Ultrasystems visited all the 
participating agencies to directly determine the shape of the experi- 
ment currently underway and to examine the data being collected. The 
results of this effort were presented to OEO in a report submitted in 
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late August of 1972. The report raised some serigus questions re- 
garding the Fnanner in which the experiment was being conducted and 
the extent, form and availability of the data collection activities 
underway as part of the experiment. Ultrasystems recommended a new 
approach to the data collection using a data collection form developed 
by Ultrasystems. Ultrasystems also recommended changes in the client 
group sample design. Both of these recommended changes were accepted 
by OEO. 

In late November 1972, OEO recwimended a revised evaluation scheme. 
This revised scheme stemmed frm continued discussions about the issues 
discussed in Ultrasystems earlier report. After considerable dis- 
cussions with OEO, Ultrasystems responded in mid-January with a revised 
work statement which addressed the evaluation design contained in the 
November letter and also proposed an optional approach. The option 
included the implementation and evaluation of a new mini-experiment 
similar to the original experiment conducted in New Jersey. This new 
experiment was to be conducted by Ultrasystems in conjunction with 
four offices of Project SER. The new experiment was approved and pro- 
vided most of the quantitative data used in this evaluation. 

To this point the above discussion has not specifically addressed in 
any detail the technical activities accomplished by the project. As 
previously stated, the original evaluation was to assess the effective- 
ness of the Cleff System in improving the quality of the job-client 
match as evidenced by referral outcomes and post-placement success; 
i.e., retention perfonnance and satisfaction. The two key sources of 
information were: 

1) The activity and client data being collected by the 
contractor running the New Jersey experiment and to 
be supplied in machine readable form to Ultrasystems. 

2) The interviews to be conducted by Ultrasystems covering 
the following classes of respondents: 

a) Agency staff and management 

b) All clients placed by the agencies 

who were given the Cleff Self-Interview 
Checklists 

c) All clients placed who were part of 
the control group 

d) The immediate job supervisor of 
each of the clients placed 

e) Personnel offices of companies who 
hired the clients from (b) above 
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Under OEO instruction, all the interviews save for the agency staff 
and management were dropped. The interviews with the staffs of the 
agencies participating in the New Jersey CJMS experiment were under- 
taken in early 1973. The results are reported in Vol. 11, Section 2. 

The analysis of the data collected during the New Jersey experiment 
was maintained in the revised statement of work. The data was col- 
lected by the contractor who ran the experiment. As noted above, the 
data collection activities were revised as a result of Ultrasystems* 
initial report. Under Ultrasystems^ original contract, all the data 
was to be supplied In machine-readable form. Along with the Ultra- 
systems' revisions to the experimental sample design and the design 
of a data collection form, Ultrasystems also took on the task of 
keypunching all the data being supplied. Whereas the original con* 
tract specified that Ultrasystems would receive the entire data base 
by October 1972, the data was not completely collected and sent to 
Ultrasystems until the Spring of 1973. The results of this data 
analysis did not provide much in the way of quantitative data for 
use in evaluating the CJMS. This is especially true as regards to the 
data required for performing a comparative retention analysis and 
more Importantly a retention analysis relating Cleff match scores to 
subsequent placement retention. The analyses are presented in Vol. 
n. Section 3. 

The basic objective of the SER mini -experiment was to obtain suffi- 
cient data to analyze the relationship betv/een the CJMS client-job 
match scores and the on-job retention for those clients placed by 
the form participating SER offices. The goal was to obtain at least 
six months' r:;teiiti on data on 150 clients. Each of the clients for 
whom retention data was obtained had taken one or both SICL's and 
each job was profiled using the Job Outline Check List and the Job 
Card Sort. Job seekers at each of the four SER offices were pro- 
filed prior to placement. The jobs were profiled after the pre- 
viously profiled job seeker began work. 

Activities related to the SER mlni-experlinent began in February 1973. 
After the mini-experiment concept was defined, the required personnel 
were trained in the administration and interpretation of the client 
self -interview check list and the resultant client scoring. Adminis- 
tration of the client checklists to SER enroUees and new applicants 
began in late March of 1973. As planned, the administration of the 
checklists continued until the end of June 1973, at which time 571 
client checklists had been completed. The activities related to ad- 
ministering the job profiles for those jobs that SER clients who took 
the Cleff booklets were placed in began In June 1973. 

A total of 110 job profiles were obtained from 100 unique employers. 
These 110 job profiles corresponded to 142 SER client placements. The 
difference is due to placefnent of more than one person Into the same 
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type of job at the same employer. The earliest hires occurred in the 
first two weeks of March 1973. The last hires occurred in early 
October 1973. Retention follow-ups were made at four time points: 
December 31, 1973; February 28, 1974; Kay 31, 1974; and December 10, 
1974. Thus, the minimum possible retention period was 14 months and 
the maximum was 21 months. 

The SER mini-experiment provided quantitative data used in analyzing 
the predictive validity of the COMS using retention time as the cri- 
terion measure. No other criterion measures, such as job satisfaction 
or job performance, were utilized. 

The Statements of Work for the New Jersey/Cleff experiment and its 
evaluation are presented in Appendices C and D. For more details 
regarding the evaluation methodology, the reader is referred to 
Section 4 of Volume I. 
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2.0 FINDINGS AND CONCLUSIONS 



The results of the evaluation and analysis of the Cleff Job Matching 
System led to the following two conclusions: 

1) The Cleff Job Matching System provides useful measures 
indicating the retention potential of people^job matches . 

The Indices provided by the CJMS are significantly related 
to job retention. In particular, the results show that the 
Job Match Index is significantly related to the probability 
that the placed individual will remain employed through the 
initial period of employment, I.e., over the first 120 days. 
The results also show that it is the Client Adjustment Index 
which relates to long-term retention, i.e., employment lasting 
over 200 days. 

2) The Cleff Job Matching System provides information about job 
seekers that is useful in assessing individuals and providing 
vocational counseling . 

The first conclusion stems from the analysis of the relationships between 
the CJMS client- job scores and on-job retention obtained from the SER 
mini -experiment. The second conclusion stems from Interviews conducted 
with Manpower and Vocational Rehabilitation personnel In the eight 
agencies that participated in the New Jersey CJMS experiment. 

It is Vltraaystems' opinion that the Cleff Job Matching 
System should be aonsidsred for use in ths labor exohcmge 
aatijfities of the U.S. Emplcnfment Service^ CETA prime 
sponsors, md other mcoipower programs. 



2.1 , , FINDINGS: VALIDITY OF THE CLEFF JOB MATCHING SYSTEM 

• AS A PREDICTOR OF RETENTION 

The predictive validity of the Cleff Job Matching System was analyzed 
using the resultant job retention of individuals who had, prior to place- 
ment, taken one or both of the Self-Interview Check Lists. The Cleff 
system's job profiles were obtained from the respective employers (I.e., 
where the individuals had been placed) after the individual was hired. 
In some cases, these job profiles were obtained after the person had 
terminated. No criterion measures other than job retention were obtained. 

Two separate types of analyses were undertaken to determine the predictive 
validity of the CJMS, using retention as the criterion. These two types 
of analyses are as follows: 
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• statistical analyses of the differences between client 
and job-match indices for client groups obtained by 
grouping those clients who worked less than or equal 
to "X" days and those who worked longer than "X" days. 
The analyses were done for five values of "X"; e.g., 
30, 60, 90, 120, and 199 days. In addition, the same 
analyses were done between those clients who terminated 
prior to December 10, 1974 and those who were still 
working as of this date. 

These analyses are referred to as the Truncation of Time 
Worked Analyses. 

• Multiple regression analyses using days worked as the 
dependent variable and the client adjustment and job match 
Indices as the independent variables. Two analyses were 
done, one of which involved the 71 people who took both 
SICLs and who terminated prior'to December 10, 1974. The 
other one utilized all 110 cases where both SICLs were taken 
and assigned a random future termination time to the 39 
people of this group who were still working as of Oecember 10, 
1974. In addition, the effect of the restriction on known 
days-worked-till-termination was estimated, using a statistical 
methodology. 

fhese two analyses were initially done independently of the reason 
for termination. Th(S first analyses were then redone considering 
the termination reasons. 

Before presenting the results of these analyses, the following points 
are presented as a brief review of the key aspects of this part of the 
evaluation: 

1) The retention of the 142 people included in our sample was 
obtained by telephone contact with the respective employers. 
These telephone contacts were done at four time points, i.e., 
December 31, 1973; February 38, 1974; May 31, 1974; and December 
10, 1974. 

2) The employer was asked if the person was still working and if 
the response was No then the date tenninated and the reason for 
the termination was obtained. No questions were asked regarding 
job or salary changes. 

3] The sample of 142 people consisted of one group of 110 people 
who completed both SICL booklets and 32 people who only completed 
the Preferences (Likes and Dislikes) SICL. 
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4) As of the last retention followup (December 10, 1974), 
94 of the 142 people had tenirinated (66.25S). Twenty- 
eight terminated after working 30 or less (calendar) 
days (approximately SOS of all the terminations), and 

47 people terminated after working 90 or less (calendar) 
days (50% of all terminations). 

5) The 142 people were hired into their respective jobs over 
a seven-month time period, i.e., from early March 1973 to 
early October 1973. Thus, the maximum and minimum retention 
periods could be from approximately 21 months to 14 months, 
respectively. 

6) No information was obu^ined from the people themselves nor 
was any other data regarding their job performance (other than 
their retention) obtained. 



2.1,1 Truncation of Time Worked Analyses 

Tables 2-1 and 2-2 show the results obtained from the analysis of the 
observed retention (measured in calendar days from the date of hire to 
the date of termination) of the i!2 individuals and the scores (indices) 
obtained from the Cleff Job Matching System. Table 2-1 shows the mean 
indices scores for client groups who worked less than or equal to X days 
or greater than X days . The values of X (i.e., job retention time) 
selected for the analysis are as follows: 

A) 30 days 

B) 60 days 

C) 90 days 

D) 120 days 

E) 199 days 

Table 2-2 shows the mean indices scores for those clients who terminated 
prior to the last retention followup point (December 10, 1974) and those 
who were still working as of that time. The average length of time 
worked for the 94 individuals who terminated prior to December 10, 1974 
was 127.8 days. The average length of time worked for the 48 individuals 
who were still working as of December 10, 1974 was 548.4 days. 

The data shown in Table 2-1 indicate that there is a statistically sig- 
nificant relationship between the Job Match Index and the probability 
that the person will remain employed over the initial 120 calendar days 
from the date of hire. 
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TABl Pi 2*1 AHALVSIS Of CUff WKn SCOfleS versus length of T1H£ 
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TABLE 2-2, anAIYSIS OF SCORES (INDICES) FOR THOSE PEOPIE STIll 
WORKING AS OF DECEH8ER 10, 1974 AND THOSE WHO 
TERMIHATEO PRIOR TO THIS TIME 



Status as of 



12-10-74 




CAI 


CD I 


JMI 


JDI 


All 


7 


29.1 


1459,3 


26,1 


2201,5 


Terminated 


s 


37,3 


786,1 


28,9 


333,4 




n ■ 


71 


71 
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94 
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7 
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2072,1 


Stil 1 
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35,3 
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27.0 
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39 


48 


48 




t 


3.11 
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1,75 
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(NOTE: See Table 2-1 for significance codes.) 
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The optimum level of truncation for the JMI appears to occur at about 
60 days. As can be seen in the two tables, the value of "t" for the 
t-test for the JMI is maximum at this level. Also, it is the only 
level at which all 4 indices are judged simultaneously to be signifi- 
cantly different. 



The CAI and CDI indices appear to pass through an initial optimum also 
at 60 days; however, there is also a long-tem effect which becomes 
even more significant at 199 days and is maximal at the time of the 
last employment status check. This is more clearly seen by the t-tests, 
which compare all "terminated" versus all "still working" as of 
December 10, 1974, as shown in Table 2-2. 

All of the people who terminated prior to this time had an average CAI 
of 29.1 compared to an average of 51.8 for those still working. The 
JMI comparison, although significant, had a much smaller difference 
(26.1 versus 34.9 for the still-working). 

FINDINGS 

The data indicates that although both CAI and JMI 
are related to job retention, it is the CAI which 
relates to tong-temremployment potential while the 
JMI is the important measure of surviving the initial 
period of employment, particularly during the first 
three months. 



2.1.2 Multivariate Regression Analysis 

In order to further delineate the relative long-term effects of the CAI 
and JMI on job-retention time, the following multiple regression analyses 
were conducted using the final set of retention data obtained as of 
Decanber 10, 1974. 

1) Using only the 71 cases where the termination time was known 
(people had terminated) and the people involved had taken 
both SICLs. 

2) Using all 110 cases where both SICLs were taken with those 
"still working" assigned a random future termination time 
based on dn estimated exponential termination time delay 
function. 

The analyses of variance resulting from the above regressions are 
presented in Table 2-3. 
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TABLE 2-3. ANALYSES OF VARIANCE FOR RETENTION TIME MULTIPLE 
REGRESSIONS 

Y = (days worked)^^^ 
X, 



'1 



CAI 
JMI 



(1) Using I TEt^I NA TION PATA Op] (N = 71) 



Degrees of 
Freedom 
SOURCE df 

Xi [CAI] 1 


Sum of 
Squares 
ss 

1.48 


Mean 

Square 
ms 

1.48 


F 
1 
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1 


24.89 


24.89 


1 
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68 


2408.19 


35.41 






Total 


70 


2434.56 









MULTIPLE CORRELATION COEFFICIENT 



0.104 



(2) Using ALL DATA (N = 110) 



Note: "Still-Working" Temlnations Randomly Assigned Calendar Days 
Employed To Temlnation From Exponential Time Decay Function 



SOURCE 


df 


ss_ 


ms 


_F 


X^ [CAI] 


1 


1224.28 


1224.28 


9.94 


Xg [JHI] 


1 


357.79 


357.79 


2.91 


Res id 


107 


13178.54 


123.16 




Total 


109 


14760.61 







SIG.' 



++ 



MULTIPLE CORRELATION COEFFICIENT 



R 0.327 



♦Significance Code: 

- = Source of variance NOT significant 
+ = Source of variance SIGNIFICANT @ 95% Confidence Level 
++ - Source of variance SIGNIFICANT @ 99% Confidence Level 
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It was not considered to be statistically appropriate to include the 
"still-working" times with the "termination" times in a composite 
regression analysis due to the fact that the resulting distribution 
of days worked is being bounded on the upper end by the date when we 
did our last foUowup, 

In an attempt to overcome this problem, use was made of an exponential 
life-testing model. The job-retention times were seen to behave more 
or less in the same way as a life-testing decay. Thus, an exponential 
time decay function was fit to all 94 cases who had terminated. The 
basic exponential distribution is given by a probability density function 
having only one parameter which is termed the "mean life" or the "mean 
time between failure." One obtains the estimated value for this parameter 
by taking the total calendar days worked by aVl_ clients (both terminated 
and still working) and dividing this by the total number of terminations 
observed. The estimated value was 407.8 days per termination. Using 
this value, one then estimates the cumulative exponential decay function 
and determines the probability that a client, in this case the clients 
still working as of December 10, 1974, will have been employed the length 
of time that they were observed to be employed. Multiplying this proba- 
bility by a random number^ one obtains a "random" point in the total . 
remaining probability scale for total time worked, which is greater than 
the observed time worked for those clients still working. One then ob- 
tains the random future termination time, which is assigned the 
still-working client as an estimate of total days employed until 
termination. (The reader is referred to Section 5.4,4 of Part II 
for a more complete description of this approach.) 

Using the random temination times for each of the "still-working" 
cases, the second regression was conducted using the resulting 
110 case composite groups, i.e., the actual terminations and the 
estimated. 

As a further measure of the effect of "restriction" on the first 
data set, which used only the j6bf\*.enn^^flitfid c9$es, an "unrestricted" 
estimate of the correlation was perfomed using a correction formula. ^ 



J.R, Guilford, fmdamental Statistios in Psychology and Education, 
4th Edition. New York: McGraw-Hill, c. 1965, pp. 341-45. (See 
Section 5.4.4 of Part II for a more complete description.) 
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2.1.3 Findings of the Analysis of the Reason for 

Termination and the Relationships of the Cleff 
System Scores to Retention 

As stated earlier, the analyses just described were done inde- 
pently of the reason for termination. Table 2-4 shows the 
termination reasons* given by the employer, and the distribution 
of calendar days worked by the people terminated for each reason. 

Examining the reasons for termination* one sees that there are 
Frtany cases in which the exact reason is not discernible. As an 
example, one sees that the largest category of terminations by 
reason is "Quit No Reason Given" {i.e., there were 17 terminations 
in this category, which is 18. U of the total of 94 terminations). 
In addition, there were two (2) terminations for which no reason 
was given by the employer (labelled INA in Table 2-4. » line 13), 
and there were 10 terminations for which the reason given was 
either Voluntary Termination or Terminated By Compaoy-No Reason 
Given. Thus, there was a total of 29 terminations (30.8% of the 
total terminations) for which the "real" reason is not clear. In 
other words* the reason for these terminations could have been the 
same as another reason listed; i.e.* Quit-No Reason Given could 
have been a reason such as Didn't Like Job or Quit To Go To School 
or Quit To Gat Married, etc. 

In addition to this problem, there is of course the question of 
the real validity of these reasons. As stated, these reasons were 
given by the employer. Since the people themselves were not con- 
tacted, no check on the reason is available. One should note that 
whereas also obtaining a reason for termination from the person 
would be a great help, it is* in Ultrasystems' opinion, likely that 
in many cases the reason given by the person will differ from that 
given by the employer. As an example, the employer reason Did Not 
Qualify might be contradicted by the person giving a reason such 
as Didn't Like the Job or The Company or Their Supervisor. That 
there would have been a difference in perspective between the two 
respondents is* as stated, our opinion, and is obviously not sup- 
ported by any data we obtained. 

Aside from these two issues, the reasons for terminations given 
also show that there were two people who never reported for work, 
four who were terminated when the business closed, and seven who 
terminated because of lack of work (or financial problems), and 
four who were hired on a temporary basis. 
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TABLE 2-4*: - TERMINATION REASONS AND DISTRIBUTION OF DAYS 
EMPLOYED BY SUCH REASON 





Line 




Ultra 


Total 






Calendar Days Employed 






nuinucr 


1 errni nac i on K6aSon 


LOQe nOS* 


Term. 




J 1 - 


61" 91- 


IZl* 




300' 


Ann coo 












O 'lO 


60 


90 120 


199 


299 


399 






1 
1 


UlQ Hoc IjUallTy 




B 


4 




2 




1 


1 






2 


Not Dependable 


16 


3 






1 1 






1 






3 


Didn't Like Job 


17 


1 


1 
















4 


Personality Problem 


30 


1 








1 












nDSccnccci Sin Ano/ur 
























par 11 


7 R 9ft 


£ 

V 


9 


9 


1 








■ 




g 


Oil 1 DaAcnn H.^ uah 


1^ 
1 o 


1 7 


7 


% 


1 9 




1 
1 








7 


Unl iintfl l^v Tp>^^ nA t i nn 
vuiuiivorjf ictiiiiiiaiiiuii 




£ 










1 
1 


1 
1 


1 






1 ct 111 1 iio iiCti vjf v#mit|jaiijf** 

No Reason Given 


26 


4 


2 




1 








1 


ft 


Q 


never i\c|jvr iicu r ur nur k 


1 
1 


9 


9 
















10 


Business Closing 


5 


4 


3 














n 

1 1 

12 


iu**vi 1 LauK ui HUrlv 
v#uiiipai|/ naa r 1 ndiiC Id 1 

r r Ul>l 
INA 


A 

1 


M 

1 
1 

9 


1 


1 
1 


1 9 






1 
1 






14 


Temporary Hire 


90 


A 

H 








*\ 
c 


c 








1 9 


km J D A^ ^ AM in ^ 

rouna oexcer ood 


1 1 








1 






1 
\ 


1 
1 




Ifi 


mi It lu uu 1 u ocnoo 1 


9K 


A 




1 
1 




1 


1 




1 
1 




1 7 


Henc inco uivn ousi ness 


1 7 


1 
1 




1 
1 














1 R 
1 o 


Uam^ On Da a n a 1 1 a A i« a 

Henc un rersona i Leave* 
Never Came Back 


21 


1 






1 












19 


Went On Personal Leave* 
Took Too Long Coining Back 


IB 


2 








1 


1 








20 


Personal Reasons 


6 


5 


2 






2 


1 








21 


Left Area 


14 


' 3 


1 






2 










22 


Quit To Get Married 


10 


4 


1 


1 




1 










23 


Pregnancy 

No Driver's License 


29 












1 








24 


2 






1 














25 


Language Problem 


31 










1 










26 


Violated Probation 


23 








1 
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Falsified Birth Cert. 


3 






1 
















Total 




28 


10 


9 7 


20 


9 


6 


3 2 
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• FINDINGS 



The predictive validity of the CJMS Client Adjustment Index 
in measuring the long-term retention probability of client- 
Job (Ttatches was not affected by the cases excluded. The 
results obtained using all the data independently of termi- 
nation reason for the predictiveness of the Job Match Index 
in the short term were significantly affected by the cases 
excluded. 

In Ultrasystems' opinion, the exclusion of the four clients 
who were terminated when the business closed the plant where 
they were employed is the clearest set of exclusions. The 
results obtained on this case show that the short-term 
predictive validity of the JMI was not significantly reduced. 

The termination reason data obtained by Ultrasystems is not 
sufficent to enable one to disaggregate the data base into 
terminations based on their degree or level of job-relatedness. 



2.2 FINDINGS: OPINIONS OF USERS OF THE CLEFF JOB MATCHING 

SYSTEM 

Interviews were conducted with the pt^ofessional staff and management 
of the eight agencies that participated in the New Jersey Cleff ex- 
periment. One of the objectives of these interviews was to obtain 
the attitudes and opinions of these CJMS users regarding the useful- 
ness of the system. This section summarizes the findings obtained. 

The eight agencies which participated in the New Jersey experiment 
consisted of three community-type agencies and five Vocational Reha- 
bilitation (VR) offices of the New Jersey Rehabilitation Commission, 
These agencies are listed below: 

Coimiunity Agencies 

* 

1) Business and Industrial Coordinating Council (BICC) 

2) Typing and Office Preparatory Skills Training Program (TOPS) 

3) Field Orientation Center For Underprivileged Spanish (FOCUS) 
Vocational Rehabilitation Agencies 

4) Atlantic City Vocational Rehabilitation (AVCR) 
{Cape May, New Jersey Outreach Office, CMVR) 
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5) Atlantic City Model Cities (ACMC) 

6) Jersey City Vocational Rehabilitation (JCVR) 

7) New Brunswick Vocational Rehabilitation (NBVR) 

8) Perth Amboy Model Cities (PAMC) 

(Diagnostic and Employment Center; State of New Jersey 
Rehabilitation Commission) 

The abbreviated title for each agency is used in presenting the 
findings. The Atlantic City Model Cities (ACMC) and the Cape May 
Vocational Rehabilitation (CMVR) are actually outreach offices of . 
the main Atlantic City VR office. 

Response to the system in all agencies visited was strong and it 
appears that, conceptually, CJMS has much to recommend it. How 
ever, mistakes were made in its introduction and operation which 
placed constraints upon its acceptance. Where these constraints 
did not work against acceptance,.. use was made of CJMS. In some 
agencies, however, these constraints acted to obviate acceptance 
which in turn caused the user staff to dwell on these constraints 
to the detriment of the system. 

Favorable opinions regarding the usefulness of the CJMS were 
obtained from staff at BICC, TOPS and the Atlantic City VR offices. 
These agencies accounted for over BQ% of the 2,600 experimental 
group clients from the New Jersey Cleff experiment. 

The responses obtained from the other Vocational Rehabilitation 
agencies were generally negative. It is Ultrasystems' opinion 
that problems associated with the manner in which the CJMS was 
introduced and technical assistance provided hindered the accept- 
ance and effective use of the CJMS by the other Vocational Reha- 
bilitation agencies. FOCUS did not make much use of the CJMS and 
hence staff personnel did not have any meaningful opinions of the 
system's effectiveness. 

The following subsections present some of the key findings from 
each agency obtained from the field interviews. Where appropriate 
brief descriptions of each agency's operations are given. The 
Vocational Rehabilitation agency personnel responses are generally 
discussed as a group with the exception of the responses obtained 
from the Atlantic City offices. 

The responses are presented in a narrative form. 
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2.2.1 



Findings From Interviews Conducted At BICC 



All the BICC personnel interviewed* including several former 
counselors no longer associated with the agency due to budget 
cutbacks* had very favorable opinions about CJMS. The counselors 
at BICC used the CJMS as a basis for selecting people for job 
referrals and for training programs that BICC conducted. The coun- 
selors used SICL results in counseling sessions and felt that the 
information provided by the SICL considerably shortened the time 
required for counseling. CJMS was used as a diagnostic tool and 
was thought to be more useful than the Kuder or Minneapolis. It 
helped the counselors reduce categorization errors such as occur 
when a client neglects to tell the interviewer of some significant 
past experience or when a sing'e DOT code is assigned to an individual 
with a broad work experience. 

In one instance* BICC administered the SICL at a nearby high school. 
The students were not interviewed beforehand and the SICL was ad- 
ministered en masse. BICC stated that the school found the SICL 
to be accurate and useful for providing the students vocational 
counseling. 

BICC counselors felt that use of CJMS resulted in a significant , 
amount of time saved in the counseling process* since it reduced 
the time required for fact finding and helped the counselor set 
client service objectives fairly soon in trie counseling process. 

BICC was the first of the agencies to use CJMS. They were instru- 
mental in introducing CJMS to OEO and participated in at least two 
revisions to the system. BICC also appeared to have received the 
njost attention from ADP-PDS in the form of technical assistance. 
This history could have given BICC alnjost a proprietary interest 
in seeing CJMS accepted. 

BICC administered more SICL's than all of the other seven agencies 
combined* accounting for almost 60 percent of all SICL's in the 
experimental group. In addition* BICC was the only agency to 
JOCL jobs and to establish a CJMS job bank. It placed more people 
from the experimental group into jobs than any of the other agencies 
(approximately 42 percent of the total). 



2.2.2 Finding's From Interviews Conducted At TOPS 

TOPS provides MOTA training in office and secretarial skills and 
counseling and job development services for its students. Since it 
is primarily a training institution* services are <?stricted to 
those accepted for training and are primarily training-related. 
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TOPS has used the CJMS to supplement Us regular screening process; 
viz., applicants for training are interviewed, tested to detennine 
educational level {Metropolitan Achievement Test) and interest (CJMS), 
and accepted or not accepted for training. Persons referred by CEP 
are accepted and assigned'training regardless of findings during 
screening as a result of contractual obligations. 

TOPS felt they could tell from the SICL whether the person referred 
by the CEP was motivated sufficiently to complete the training {in 
addition to being suitable for such training). Although TOPS tried 
to have the CEP withdraw or change what the SICL indicated was an 
inappropriate referral, they were unsuccessful. TOPS felt this CEP 
refusal to change or withdraw the referral was somewhat burdensome, 
since the individual generally dropped out before completion of 
training. 

Once an individual was accepted for training, TOPS provided counsel- 
ing services designed to help the client continue in the training 
and to help the client prepare for a job in the area in which he 
was being trained. The counselor felt that the SICL results, 
especially the "People-oriented" parameters, were very useful in 
counseling at this point. Another benefit cited for CJMS was that 
it forced TOPS to improve their training. Since CJMS helped TOPS 
pick the best people, the training standards had to be»raised a 
corresponding amount, 

TOPS staff stated that they felt very definitely handicapped with- 
out CJMS and, if they had the money, would start using it again. 



2.2.3 Findings From Interviews Conducted At ACVR 

The Atlantic City offices have been able to make extensive use of 
CJMS, although it appears that the counselors are still probing 
to determine the limits of the system. Cape May counselors, for 
example, can interpret the CJMS results but feel they need more 
experience interpreting in order to be able to derive the maximum 
amount of information from the results. Nevertheless, they have 
found that CJMS results provide them with a direction in counseling. 
This helps them get into effective counseling more rapidly, since 
it speeds up the fact-finding portion of the counseling period and 
helps avoid some counseling blind alleys. In addition, the Cape 
May counselors feel the CJMS results serve as a good tool for 
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determining a need for a psychological or psychiatric examination. 
^rote that the CJM5 is not used as a psychological diagnosis but 
rather as an indicator that such diagnosis may be warranted. 

Similarly, ACVR and ACMC counselors feel that CJMS gets them into 
counseling faster by providing a shortcut to fact finding. These 
counselors feel CJMS has some advantage over psychological exami- 
nations in that it breaks down client/job-related characteristics 
into more specific areas. This enables them to establish counsel- 
ing and vocational objectives faster. They find the CJMS especially 
useful for working with the mentally retarded, since they feel that 
standard I.Q. tests provide meaningless results, whereas the CJMS 
helps this client express his interests and vocational strengths. 

One counselor stated that she used the SICL as a quick LQ. test 
to determine if a client should be sent to college. A psychologi- 
cal test is still conducted since this provides an I.Q. score which 
must be used to justify the decision for the record. In general, 
Atlantic City counselors liked the speed with which they could 
obtain usable results. They could get SICL results back in the 
same day or within a few days, whereas it could take up to a week 
before a client could meet with a psychologist and up to three weeks 
before the psychologist's report could be received. 

While Atlantic City counselors did not notice a reduction in time 
required for counseling, some did think that CJMS helped them to 
set counseling objectives sooner than they might have without CJMS. 

The Atlantic City Vocational Rehabilitation agencies had elected 
to continue using CJMS and had budgeted funds for CJMS. This was 
th*e only agency that had elected, at the time of the interviews, 
to purchase the system and allocate funds for its operation. 

2.2.4 Findings From Interviews Conducted At FOCUS 

FOCUS does not appear to have made much use of the SICL results. 
They called in the respondent and went over the results with him. 
They found most people ;*ho took a SICL anxious to find out the 
results but were unabld to make much use of the results. Lack of 
facility in English and no training stipends limited clients in the 
training and jobs they would accept or could expect to get. FOCUS 
felt that the SICL was useful in helping some clients make up their 
minds for the future but, due to the limited resources available 
to them (or FOCUS), the SICL was not applicable. Most referrals 
of clients to jobs w6re made on the basis of the client's prior 
experience, not the SICL. 
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2.2.5 



The Vocational Rehabilitation Agencies: General 
Observations And Findings From The Interviews 



At the Vocational Rehabilitation agencies the counselors assess 
the individual's eligibility for service and determine what services 
to offer the client. The counselor may require the client to under- 
go any of a number of diagnostic services in order to ascertain 
appropriate remedial services, CJMS was accepted by the agencies 
in this study as another diagnostic service to add to their repertoire. 

The counselor thus could select the CJMS SICL as one of the diag- 
nostic services to offer a client. The SICL could fee administered 
at any time during the client's tenure of service at the counselor's 
discretion. The counselors are therefore the key to the use of 
CJMS in the Vocational Rehabilitation offices. In Ultrasystems' 
opinion, AOP-PDS initially had problems communicating to the Voca- 
tional Rehabilitation offices selected to participate in the experi- 
ment what the experiment was about and whom it was to serve. About 
two months after the installation of CJMS, the Vocational Rehabili- 
tation offices held a meeting to clarify these issues, and the New 
Jersey State Vocational Rehabilitation Director instructed the office 
managers to use the system. No additional staff was allocated to 
administer or process the SICL's. This caused objections, since the 
offices already felt that they were understaffed, ADP-PDS conducted 
at least three training sessions for the staff of these offices, but 
failed to convince many of the counselors of the value of the system 
as compared to other diagnostic services then available. These dif- 
ficulties, plus related difficulties discussed below, posed a severe 
handicap to CJMS. The handicap was severe enough in one office 
(Jersey City) that, except for some experimentation, CJMS was not, 
used. 

Another limitation to the use of CJMS in the Vocational Rehabilita- 
tion offices lies in how the offices perceive their mission. They 
art* not anployment offices and have no full-time job developers. 
Job development is usually performed by counselors through their own 
contacts or through referrals to other agencies such as the Employ- 
ment Service. Individuals entering with medical problems tend to 
get services designed to alleviate the specific diagnosed complaint. 
Unless employment requires special considerations on the part of the 
employer, as with mentally retarded persons, or is considered part 
of required treatment* as with drug addicts » or is easily attainable, 
the client is not usually given much help in finding employment. 
The client in this case will be referred to the Employment Service 
or returned to the agency which referred him in the first place. 
Thus, one of the offices felb that most of its caseload "were medical 
problems and not employment problems," 
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All of the Vocational Rehabilitation offices, with the exception 
of the Perth Amboy Model Cities (PAMC) office, contract with outside 
professionals for diagnostic services. PAMC alone has its own exten- 
sive vocational evaluation unit and, as such, was considered to be 
a pilot project of New Jersey Vocational Rehabilitation. 

A psychological or psychiatric diagnostic report usually consists 
of several pages of narrative describing tests given, the test results, 
results of personal interviews (including some background data which 
may or may not have been uncovered by the counselor), descriptions 
of clients' reactions during the examination period, summarizing 
conclusions, and suggestions for a future course of action. The 
SICL provides a set of experi6;.ce, preference, and combination scores, 
measures of correlations and differences for these scoras and, if 
requested, matches to jobs or clusters with appropriate scores and 
measures of correlation and difference. A manual is provided by 
ADP-PDS which defines each of the scores and the measures of corre- 
lation and difference and provides information on how to use these 
scores and nieasures. The clusters accompanying a SICL match suggest 
a number of jobs the client might do best. 

In the psychological or psychiatric narrative, much of the interpre- 
tation of tests and correlation of data has been done for the coun- 
selor by the examiner and this information is arranged so as to 
support the examiner's conclusions. Each of the Vocational Rehabil- 
itation offices, except for Perth Amboy, exprcissed dissatisfaction 
with the presentation of SICL results. 

The senior counselor at the Jersey City VR felt that compared to 
the psychological evaluation, CJMS "was a joke ... (you) can get 
all the information that (CJMS) gets from the interview." Both the 
New Brunswick and Cape May VR offices complained of the time required 
to interpret the SICL. Each of the offices stated that it required 
time to learn how to interpret the SICL, experience in the, job mar- 
ket in order to be able to relate the clients to the job market, 
knowledge of the jobs actually available to their clients, and 
experience with CJMS in order to understand CJMS well enough to use 
it as a diagnostic tool. 

Ther Jersey X^ty Vocational Rehabilitation (JCVR) office felt that 
CJMS was inferior to the quality and sophistication obtainable from 
their consultants. NBVR similarly saw no advantage to using the 
CJMS because of the expenditure of their own time to administer the 
SICl, the quality of interpretation, and the time required to obtain 
results when they mailed the SICl for scoring, JCVR stated that 
they gave SICL's and psychological to two clients and that the 
psychological s were "much more revealing and informative." 
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Atlantic City VR had reduced the problem somewhat by hr.ving the 
AOP-PDS technician chart the results. The charting is simply a 
clarification on the presentation of SICL results and counselors 
still expressed preferences for the psychological narratives. 

Perth Amboy MC presented the results of the tests performed by its 
evaluation section on an overall score sheet which includes the 
evaluation of the results of all the tests including the SICL. 
The majority of counselors and evaluators felt insecure in inter- 
preting the SICL's since they don't fully understand the specific 
meaning of the parameters or statistical base for determining the 
validity of these parameters. PAMC evaluators and counselors feel 
that CJMS may be somewhat useful for inexperienced persons with no 
idea of direction, but that they would rather spend the time inter- 
viewing the client. 

Some counselors questioned how the SICL could be interpreted at all 
for some clients and gave as examples clients whose point of refer- 
ence was so low that they had never heard of some phrases. The 
counselors agreed that lack of knowledge might indicate lack of 
interest, but they pointed out that each page of the SICL contained 
many such phrases. 



2.2.5.1 Cwnments Regarding Acceptance Of The CJMS By Vocational 
Rehabilitation Personnel 



The objective of the interviewing was to determine how CJHS was used, 
what impact it had on staff/client relationships, and to gain insights 
into how the system could be used and whether these uses appeared to 
be effective. The purpose was to evaluate the system and not the 
agencies using the system or the services being supplied* 
These objectives held up in the interviews with the community agencies 
but collapsed in the Vocational Rehabilitation agencies as it be- 
came evident that there were problems external to CJMS that were 
having deleterious effects on CJMS usage. 

It appears, for example, that CJHS was imposed upon the Vocational 
Rehabilitation agencies in the experiment and that they were virtu- 
ally ordered to use CJMS by their State- Director. We have no direct 
evidence to support this conclusion other than cautiously worded 
statements from staff members which seem to imply this. In addition, 
it appears that some of the personnel responsible for introducing 
CJMS acted boorishly and tactlessly with Vocational Rehabilitation 
staff persons and that the manner in which technical assistance was 
provided only made problems worse. 
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In one office staff members complained vehemently of the treatment 
they received from ADP-POS staff members. They told of 1) brisk 
responses from ADP-PDS to their questions; 2) calling AOP-POS offices 
in New York and not being able to get someone who could answer their 
questions or of not receiving answers after being told that someone 
from ADP-POS would call back with an answer; 3) receiving CJMS mate- 
rials with what they considered inadequate instructions as to their 
use and then not being able to obtain clarification in writing or 
through the personal appearance of a Technical Advisor. Staff mem- 
bers conceded that these experiences possibly had a negative effect 
on their opinion and ability to use CJMS. 

In Atlantic City, staff members spoke freely of CJMS but were reluc- 
tant to discuss the personality conflicts other than to acknowledge 
that they knew of the existence of these problems. The staff did 
mention difficulties in getting technical assistance with respect 
to CJMS until they had actually purchased the system at the end of 
the experiment. 

It is believed by Ultrasystems that the conflicts caused by these 
problems hindered the acceptance and effective use of CJMS by the 
Vocational Rehabilitation agencies. Why Atlantic City was not 
affected by these problems is unknown. What the Vocational Rehabil- 
itation agencies' experiences with and attitudes toward the system 
would have been had these problems not existed is unknown. 
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2.3 GENERAL OBSERVATIONS ANO OPINIONS: USE OF THE 

CLEFF JOB MATCHING SYSTEM IN MANPOWER OPERATIONS 

The observations and opinions to be presented In this section are 
concerned with the use of the CJMS In manpower operations. They 
are based on UUrasystems' observations of the CJKS being used In 
manpower agencies and on our own experiences derived from the 
SER/Cleff experiment. The topics to be discussed are as follows 
(there- Is no relative Importance Inferred by the order): 

1) The time needed to complete the checklists. 

2) The self-completion aspects of the SICL and the ilOCL. 

3) The use of the Cluster Register. 

4) The Instructions and explanations available regarding 
the Interpretation of the system's scores. 

5) Other operational aspects of the use of the CJMS. 

6) The impact on agency performance and procedures. 



2.3.1 Time Necessary for Completion of Self-Interview 
and Job Outline Checklists 

Our observations and experiences Indicate that It requires between 
one and two hours for the completion of both SICL booklets and 
approximately one hour for the completion of the JOCL. There are 
no pre-set time limits associated with checklist completion. These 
estimated times are based on completing two 15-page siCLs and the 
10-page JOCL plus card sort. The times also Include giving the 
Instructions regarding the mechanics of completion. 

UUrasystems has no experiences associated with administering more 
than one JOCL at a time. There are three possibilities here, as 
follows: 1) where a number of supervisors each complete a JOCL for 
a given job at the same time; 2) where a number of supervisors each 
complete a JOCL for different jobs at the same time; and 3) a mix- 
ture of the two. It Is UUrasystems' opinion that possibility X 
(one job - more than one supervisor) Is the most amenable to group 
JOCL completion. 

UUrasystems is aware of and has observed the use of shorter SICLs, 
I.e.* 10 pages per booklet instead of 15. Me have never seen any 
data regarding the effect of partial SICL completion, nor have we 
analyzed the data collected In this project In this regard. It Is 



ERIC 



1-29 

41 



in^ortant to note that many of the SICLs associated with the SER 
experimental data base were not fully completed. A11 SICLs utilized 
In the data base had at least 10 completed groups per SICL. Ultra- 
systems is not aware of the use of shorter JOCLs nor have we analyzed 
the job profiles completed In this project In this regard. 

The above times do not Include the time needed to input SICL scores 
for conversion to dimension scores and for the calculation of the 
client and job match Indices. The times also do not Include the 
time needed to Input JOCL scores. These Input processing and gener- 
ations of output times are a function of the method of processing 
utilized. Ultrasystems did not obtain data regarding the Input times 
associated with the use of the teletype Input/output system utilized 
in New Jersey. The procedures utilized in the SER experiment require 
approximately 10 minutes for the transcription of the SICL data onto 
the Input form and approximately 12 minutes for the keypunch of same. 
The calculation of the JOCL dimension scores from the completed book- 
let and card sort requires approximately five minutes. Keypunch time 
is dependent on the amount of other job order data included. In the 
SER experimenti the job order form including the CJMS scores required 
about six minutes for keypunch. 

In Ultrasystems' opinion^ the time requirements associated with the 
use of the CJMS In a manpower operation are significant. The time 
spent by the client Is an imposition on the client. The degree to 
which people taking SICLs need to be supervised affects the direct 
costs associated with the system, Ultrasystems does not believe 
administration of SICLs needs to consume the full time of the admin- 
istrator while they are being completed, Thereforei the acbninlstrator 
could be doing some other clerical-type tasks during the SICL com- 
pletion processt but should be physically with the people taking 
the booklets. 

The time required to complete JOCLs Imposes a different operational 
problem in that It requires that at least one job supervisor give 
up one hour of paid time. In other words i part of the cost is borne 
by the employer. The average time required to complete one JOCL Is 
much greater than the one hour needed to complete the booklet. 
Obviously, arrangements must be made with the employer and the admin- 
istrator must travel to the worksite. In addltloni there are high 
Incidences of Interruptions when completing JOCLs and of the need to 
reschedule appointments because of work requirements. 

Administrating SICLs In groups requires that the group be assembled 
at one time. This may Impose further time requirements on the people. 
Assuming that one full-time SICL administrator costs $30.00 per day 
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(salary, fringe benefits, overhead), and that it takes one and a 
half hours per person to complete two SICLs (15 pages each), and 
that theStCLs are administered in groups of five, then in one seven- 
and-a-half-hour day 25 SICLs can be administered at a direct labor 
cost of $1.20 per SICL. Inputting 25 SICLs at 12 minutes per SICL 
(with the labor cost of a keypunch or teletype operator figured at 
$45.00 per seven-and-a-half-hour day) would cost another $1.20 per 
SICL. If one estimates that it requires four hours per completed 
JOCL (includes time to arrange for the interview and to travel to 
and from the worksite) at a cost of $50.00 per day (for the job 
developer), then it would cost $25.00 per JOCL. 

The above costs are all estimates. In addition, these costs do not 
include the computer processing associated with the CJMS, the cost 
of materials, travel, and the physical plant requirements. They 
also do not include the costs associated with determining employers 
to be contacted and deciding on the jobs to be profiled. In addition, 
no time or costs associated with the resolution of job profile dif- 
ferences between two or more supervisor-completed profiles of the 
same jobs have been estimated. Furthermore, the time and cost esti- 
mates do not include the time and cost associated with selecting 
clients to whom the SICLs will be administered. 

In conclusion, Ultrasystems finds that the time and costs associated 
with the use of the .CJMS are an important consideration to be care- 
fully analyzed when setting up procedures for its use. It is Ultra- 
systems' opinion that the costs are not prohibitive. One should 
note that not only are the costs just estimates but that no cost 
benefits or comparative cost benefits have been determined. Ultra- 
systems believes that validation studies associated with the use 
of shorter SICLs should be undertaken. The use of shorter SICLs 
will not only reduce the direct costs per SICL but will, more Impor- 
tantly, impose l«ss time requiranents on the agency clients. 



2.3.2 Self-completion Aspects of the Cleff Interview Checklists 

The issue of self-cotipletions of the checklists is one of the most 
important aspects of the use of the CJMS that, in UUrasystems' 
opinion, needs to be resolved. It affects not only the costs in- 
volved (due to its effect on the requirement for supervision and 
guidance) but is critical in determining the procedures to be utilized 
and the skill level required of the instrument administrators, and 
will, to sane currently unknown degree, affect the resultant validity 
of the system's scores-. 
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The problem in self-completion is due to the difference between the 
literal interpretation of the activity phrases contained in the 
checklists and the association between the activity as described 
and the same activity expressed in a different context, i.e., with 
a different noun and even a different verb. The problem surfaces 
on a given page where the respondent cannot find enough phrases for 
selection. The problem occurs more frequently with the Experience 
(Done and Not Done) SICL. 

In addition to the issue of self-completion, Ultrasystems believes 
that the following aspects of SICL administration need to be more 
clearly defined (and substantiated): 

1) The reading ability required to complete the SICLs. 

2) The relationships between the phrases as currently given 
and the cultural backgrounds of the people. In this regard, 
the instruFnents need to consider differences in dialect in 

a given language, i.e., Spanish as used in Mexico, Puerto 
Rico, Cuba, etc. 

3) The determination of the usefulness (or validity) of the 
Experience profile based on the work history of the respond- 
ent. Ultrasystems has never seen a clear statement regard- 
ing the decision criteria to be applied in the selection of 
respondents who should not complete the Experience SICL. 



2.3.3 Use of the Cleff Cluster Register 

The Cleff Cluster Register contains 19 cluster profiles obtained 
from the mathematical analysis of somewhere on the order of 525 
to 750 individual job profiles (see Section 5 of Vol. II). These 
cluster profiles provide measures of client-job matches in the 
absence of or in addition to matches to individual job profiles 
obtained via the JOCL. Ultrasystems did not undertake an analysis 
of the effect that client-cluster match scores would have on the 
relationship between retention anu the values of the job match and 
job difference indices that would apply based on the use of the 
cluster that best matcned the profile obtained for the job obtained 
by the person. 

The Cleff Cluster Register states that the individual jobs were 
collected and then sorted into the clusters such that the correla- 
tion between the specific job profile and the cluster profile would 
be on the average +.90 and not less than +.80. The 110 job profiles 
•obtained by Ultrasystems and used in the retention analysis had a 
mean cluster correlation of +.81 with 31 having cluster correla- 
tions less than +.80 (see Section 5 of Vol. II), The range was 
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from +.51 to +.96. We examined the profiles obtained and found 
that there was homogeneity between the six Itltrasystems* employees 
who administered the JOCLs and between the time when they were done 
in terms of those profiles whose cluster correlations were less than 
or greater than or equal to +.80. Whereas we did not examine the 
profiles whose cluster correlations were above or below +.8 in terms 
of the types of jobs and/or companies involved and whereas we can- 
not be certain that these profiles were all correctly administered, 
we do believe that the job profile/cluster correlations obtained 
indicate that the number of profiUs and the types of jobs profiled 
used to obtain the number of and shape of the cluster profiles in 
the register is not sufficient. In other words* it is Ultrasystems' 
opinion that more profiles are needed before cluster shapes can be 
determined that are benchmarks representing the profiles associated 
with the spectrum of low- and semi-skilled jobs. 

In addition* it is again Ultrasystems' opinion that the following 
aspects of the use of the Cluster Register needs to be more clearly 
defined (and substantiated): 

1) The mechanism by which one can determine from a non-JOCL 
job description the appropriate cluster needs to be defined 
and validated. In other words, if the CJMS is to be used 
wherein it is not mandatory that every client-job match be 
measured using the job profile obtained via the JOCL, then 
definitive instructions are needed for obtaining verbal job 
descriptions and for determining the appr<^riate cluster. 

2) In tiie same vein as 1 above* definitive instructions regard- 
ing the types of jobs to be solicited from employers based 
on client-cluster matches are needed. In other words* one 
needs instructions for assigning clusters from verbal job 
descriptions and for determining from the clusters the types 
of jobs that would* if profiled, have the highest correla- 
tion with the given cluster. 

3) Ultrasystems believes that analysis of the relationships 
between Cleff job (and client) profiles and the Dictionary 
of Occupational Titles 6-digit job codes and associated 
Worker Trait Group profiles should be undertaken. 
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2.3.4 Interpretation of the CJMS Client and Job Profiles 
and Match Indices 



Ultrasystems believes that the material it has seen regarding the 
interpretation of the system scores is Inadequate. Ultrasystems 
further believes that such material should be developed and sub- 
stantiated for use in the following different applications: 

• Job matching as done by manpower agencies involved in 
client-job referral activities 

• Selection of training programs for referral of people 
as done by manpower agencies 

• Counseling of people regarding career choice 

• Screening of job applicants as done by employers 

• Selection of jobs for which job orders are to be 
solicited from employers by manpower agencies 

The issues that need to be discussed include the following: 

• Rigidity of indices. cutoff scores 

• Interpretation of profile dimension scores in conjunction 
with and independently of the indices scores 

• Interpretation of the indices scores in conjunction with 
each other 

• Interpretation of combined* preference* or experience 
scores in conjunction and independently 

• Use of cluster match scores 

Ultrasystems hastens to point out that we strongly believe that there 
are more factors to be considered in the execution of the applications 
listed above than are provided by the Cleff system. This is not a 
criticism of the Cleff system nor should it be so criticized. The 
list of issues regarding the interpretation of Cleff system scores 
should not be construed to imply that these are the only factors 
associated with person-job or person-training matching. There are 
obviously other factors to be considered. 

The U.S. Department of Labor has previously criticized the CJMS 
because it does not include other factors (or measures) associated 
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with person-job matching. Ultrasystems considers this an unfair 
criticism, if it meant to imply that because it {the CJMS) does not 
contain these other measures, then the measures it does provide are 
either invalid* not needed, or already adequately measured. In 
other words, to criticize the system for what it does not purport 
to measure is to ignore what it is meant to do. The system can only 
be and should only be judged on the validity of the measures pro- 
vided from which the appropriateness of people-job matches can be 
inferred. 



2.3.5 Other Operational Aspects Associated With the Use 
of the CJMS 



The preceding discussion has touched upon operational aspects of 
the CJKS associated with instrument administration, score interpre- 
tation, client selection, use of the Cluster Register, and self- 
completion. The following brief descriptions are associated with 
other aspects of CJMS usage: 

1) Capture of Oata for Use In Analysis of the CJMS 

UltrasystCTis considers it of paramount importance that the 
data necessary for further analysis of the CJMS's validity 
and usefulness be obtained and anlyzed whenever and where- 
ever the system is utilized. Ultrasystems believes this 
should be done regardless of whether or not the system is 
being used in an experimental or operational manner. 

2) Inclusion of Other "Search" Variables 

Ultrasystems believes that the system would be more useful 
if it were combined directly with other search or match 
algorithms rather than be operated independently. Exactly 
what these other measures are, or more exactly, how one 
uses them to determine appropriate matches, is beyond the 
scope of this project. 

3) CJMS Computer Program Improvements 

In addition to the inclusion of other search measures and 
algorithms, Ultrasystems believes the usefulness of the 
CJMS would be improved if the computer program used in its 
operation included the multidirectional search capabilities 
that have been developed. In other words, Ultrasystems 
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believes It would be cost effective to Include the CJMS 
algorithms within the structure of an already developed 
person-job matching program which possesses the capability 
of searching client files given a job order and vice versa, 
I.e., given a client search the job files. Ultrasystens 
believes It Is a serious mistake to create an entire com- 
puter program around a given set of search algorithms. The 
technology exists for the development of person-job match- 
ing computer software capable of executing one or more 
search algorithms and of Incorporating at the user's dis- 
cretion additional search nwthodologles. In addition, the 
technology exists whereby the computer capability can be 
modularized for use with different types of Input-output 
procedures. I.e., on-line, remote batch, or conventional 
batch processing. 

2.3.6 Some Comments on the Possible Impact of the Use 
of the CJMS on Manpower Operations 

The discussion to follow, and In fact the discussions preceding 
this one. Is based In a sense on the validity of the CJMS as a 
partial predictor of the appropriateness of people-job matches. 
The conclusion reached by Ultrasystems regarding the predictive 
ability of the CJMS was based solely on the data obtained In the 
SER experiment. Ultrasystems Is fully aware of the limitations 
that must be placed on the conclusion reached given the data that 
was utilized. If the system does not provide valid measures from 
which one can, with a given and acceptable degree of confidence. 
Infer the appropriateness of people-job matches, then obviously 
any other Issues associated with Its use are Immaterial. Since 
the conclusion Ultrasystems reached was that the data obtained 
and analyzed showed validity* we have Included the previous dis- 
cussions and will proceed with this one. 

The data obtained from the SER experiment showed that the CAI was 
the measure most Indicative of the long-term retention potential 
of the person. Since this measure Is not controllable by the 
choice of jobs, from among alternatives. It Implies that the match- 
ing process is, at least partially, constrained not by the job but 
by the person. The data obtained from the SER experiment did, how- 
ever, show that the OMI Is predictive of the short-term retention 
of the job-person match. Thus, the data showed that one would 
choose from among alternative jobs the one with the highest value 
of the JMI Independent of the CAI (with all other factors assumed 
to be equal). It thus appears to Ultrasystems that the h1^/her the 
CAI is the more latitude one would have In choosing possible jobs 
based on the JMI and, conversely, the lower the CAI the more care 
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should be taken in the job selection process. The ability of a 
given manpower program to control the match process in terms of 
the level of the JMI is going to be constrained by the jobs that 
are available to it. The New Jersey Cleff experiment had a cutoff 
criteria, based on the CJMS indices* to govern the referral process. 
These criteria were that for a referral to be made the CAI had to 
be greater than or equal to +.25 and the JMI had to be greater 
than or equal to +.50. Using these criteria* one sees that one 
is immediately constrained in those cases where the CAI is less 
than +.25 no matter what jobs are available. The use of such a 
CAI cutoff criterion is thus a very serious issue. 

In addition, if only the Preference book is completed, then one 
has no value for the CAI. What one does in these cases we do not 
know. 

Figure 2-1 shows the distribution* for that part of the SER experi- 
mental data base who took both SICLs (110 cases out of 142) of the 
paired values of the CAI and the JMI. The spectrum of paired values 
has been divided into nine cells. The arrows indicate the cells 
in which entries that fall on the cell boundaries were placed {see 
Section 5 of ?art II for the complete definition of the cell bounda- 
ries). This figure shows the number of observations in each cell* 
the percent of the total observations* the number that terminated 
prior to December 10* 1974* and the percent of each cell who termi- 
nated. The figure shows that only 14 placements out of the 110 
observations (12.7%) would have met the criteria given above. In 
addition, the figure shows that 74 people out of the 110 (67.3%) 
would have met the minimum CAI criterion independent of job match 
score. Of the total 142 person-job matches* there were 30 {21. IX) 
which would have met the minimum JMI index score independent of 
the CAI. One must be very careful in interpreting this data because 
it does not imply that had the criteria liven above been used {or 
any other cutoff criteria by which at least one case would have been 
excluded) that some of these People would not have been placed . In. 
addition, it ig not definitively known how representative these 
cases are of Slk clients nor is it known at all how representative 
they are of any other populatiorT The same is true for the job 
Profiles . 

The figure does show that had the criteria been applied to these 
cases then 32.7% would not have been eligible for referral because 
of their CAI* and of those 74 people eligible for referral based 
on their CAI, that 60* or 81%, would 'lot have been eligible for 
the job they were in fact referred to"and placed into . 
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Figure 2-1. Distribution of Paired Observations: 
Client Adjustment Index (CAI) and Job 
Match Index (JMI) 
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The datji shown cannot be used to estltnate the percentage of clients 
for whom, at a given time and with all other factors equal, the 
use of a cutoff criterion based on COKS indices would have pre- 
cluded a referral from being made. 

The discussion does point out, however* that the use of cutoff 
criteria could have a significant effect and will have at least 
some effect on the ability of the manpower organization to refer 
and place people. Obviously, this is what the system is all about; 
i.e., it is supposed to provide measures that affect the job matches 
made. However, it Is important to bear in mind that the system 
with cutoff criteria is going to have differential effects among 
manpower programs based on the clients who use this organization 4nd 
the jobs available to this organization. Obviously, the relation- 
ships between clients who come to a given manpower organization and 
the jobs available to it influence the performance of this organi- 
zation Independently of the CJMS. In fact» if the CJHS does indeed 
provide valid measures, then it is useful in that it can in a quan- 
titative sense direct the organization's activities, at least as 
regards solicitation of jobs best suited to its clientele. There- 
fore, it is, in Ultrasystems' ojxriion, crucial that the use of the 
CJMS not be restricted in the conventional sense of person-job 
matching, but that it be used in conjunction with the process of 
job solicitation. Stated another way, the level and process of job 
solicitation and the types of jobs usually available are going, in 
our *.iion, to have a significant effect on the usefulness of the 
CJMS within a given manpower organization. 

One should also not lose sight of the following; 

1) The system was developed based on data obtained from a 
certain class of individuals, i.e., those who were employed 
in low- or semi-skilled jobs. 

2) The predictive criterion-related validation done using 
the data obtained from the SCR experiment cannot be used 
to conclude validity on criterion measures other than 
retention or for populations of jobs and people not 
representative of those included in the data base. 
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2.4 EVALUATION ANO EXPERIMENTATION OF THE CLEFF SYSTEM IN 

OPERATIONAL ENVIRONMENTS 



This section will discuss some factors that Ultrasystems believes 
are Important In constructing evaluation designs for use with the 
CJMS as well as with other approaches to man^Job matching. 

It is Ultrasystems' opinion that doing a comparative evaluation of 
the CJMS Is much more complex than doing some form of validation 
analysis. As stated earlier, Ultrasystems believes that It Is of 
paramount Importance that data be assembled for use In evaluation 
and validation whenever and wherever the CJMS Is being used. 

The formulation of a comparative evaluation design for use with the 
CJMS should, 1n Ultrasystems' opinion, consider the following Issues: 

1) The comparative evaluation should Include, In the design, 
comparison of both placement and post-placement performance. 
The measures postulated for the New Jersey Cleff experiment 
are. In our opinion, a good starting point. 

Without trying to presuppose the results of such an evalu- 
ation, Ultrasystems believes there Is a good possibility 
that Imprpvements In placement perfomance measures may 
not be possible to achieve. In the short term. In conjunc- 
tion with Improvements In post-placement performance. 
Obviously, the determination of the Interaction between 
these two objectives Is one of the major purposes of doing 
such a comparative evaluation. The rationale behind this 
belief Is based on the following assumptions: 

a) that Improving the person-job match (selection) 
process Is one of the keys to Improving the post- 
placement measures 

b) that the comparative evaluation Is conducted so 
that the alternative person-job matching methodology 
Is the only process that Is changed and thus dif- 
ferentiates the two groups 

c) that the use of an alternative matching strategy 
which will Improve post-placement performance Is 
going to be accompanied directly by a narrowing of 
the range of alternative matches that can be made 
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d) that narrowing the range of matches is going to 
adversely affect placement perfonnance when no other 
changes are made, i.e.* when no changes are made in 
the organizational processes to Increase the number 
and types of jobs available, the number and types of 
clients, and the exposure provided each to each other. 

Ultrasystems is aware of t.ie view that improving the quality 
of the person-job match can be considered as a way of increas- 
ing the use made of the referral organization by employers 
and people. That this could occur is true; however, Ultra- 
systems believes this takes some time to establish. This was 
the meaning of the phrase "In the short tenn." If, in fact, 
the match strategy narrows the alternatives, then one may be 
further constrained by the alternatives currently available 
before the impact of improved quality of matches can be brought 
to bear. One is in sort of a chicken-and-egg situation in 
that increased access to jobs and jobseekers may be needed 
In order for the new match strategy to really be capable of 
working and that obtaining this increased access may not be 
possible unless the improved quality of the matches can be 
demonstrated. 

If a comparative evaluation Is undertaken wherein more than 
just the matching strategies are changed, then It will be 
difficult, if not Impossible, to establish the causal link- 
ages between all criterion measures utilized and each of the 
process changes that are made. One should keep in mind that 
cwnputerized matching systems which attempt to replicate the 
matching strategies now executed in the minds of the referral 
Intermediaries are predicated on the idea that increasing 
the nuntier of alternative jobs considered will increase place- 
ment performance. Thus, they are predicated on the power of 
the coniputer to execute the logic, now done in the minds of 
the people involved, across a larger number of alternatives 
in a time period not humanly possible. 

Ultrasystens does not believe that, in fact, one can with a 
computer replicate the process by which human judgment is 
brought to bear oa'the referral process. It is our opinion 
that one is forced to quantify in some way what are not 
totally quantifiable decisions. In addition, one obtains 
total repetitiveness in the manner in which the matches are 
calculated. Ultrasystems does not want to convey the impres- 
sion that the matching strategies now used by referral inter- 
viewers are really defineable. In many ways, what really 
occuis is s(»newhat unclear. It also varies considerably 
across the personnel involved. Computerization will by 
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definition have to standardize the approach. This does not 
mean that human Intervention will not be brought to bear. 
If human Intervention does Intervene In the sense of screen- 
ing the matches chosen by the computerized search logic, 
then It is constrained by the matches that were chosen prior 
to the Intervention. If the computerized search logic depends 
on the Input of search variable values determined by the 
referral Intermediary, then It Is susceptible to differences 
in the understanding that goes into their selection. 

The computer aspects of people-job matching are not the only 
issues associated with process changes attendant with the 
use of a'( alternative match system. Increased job solici- 
tation is another key process change which, if Introduced 
along with a match strategy change, can cloud the causal 
inferences that can be drawn based on changes In the evalua- 
tive measures. 

3) The comparative evaluation of placement and post-placement 
performance has obviously as its objective the measurement 
of the effects of the use of the alternative matching 
strategy. The degree that the alternative strategy Is 
actually used Is therefore Important. One of the objectives 
of a comparative evaluation should be to obtain a measure 
of how applicable the alternative strategy really Is Inde- 
pendent of Its validity and Independent of other changes 
In process that are necessary. This Is a difficult Issue, 
but It Is Important with a system such as the CJMS, since 
the measures provided are not all that should be used In 
selecting jobs for referral. In other words. It Is our 
opinion that the use of the CJMS measures to totally direct 
the referral process is not acceptable. The degree to which 
other factors are analyzed and used to direct the match pro- 
cess conditions the causal link between placement performance 
changes and the use of the system. 

In this regard* then. It Is essential that data be obtained 
which ci'A be used to assess the degree to which the system 
Is used and. In fact> can be used. One way Is to analyze 
the referrals that were made In terms of the systems scores. 
In order to do this, one will need to know the referrals that 
were possible for a given client at the points In time when 
the client was available for referral. Ultrasystems believes 
that a worthwhile approach would be to Implement the admin- 
istration of SICLs and JOCLs for a period of time in which 
the resultant scores and profiles woulfi not be made available 
to the referral personnel. One could then analyze the effect 



1-42 



54 



that imposing the use of the system would have on the refer- 
rals that could be made and that were made. One will not 
know what the outcome of any alternative referrals would have 
been but one could obtain some measure of the degree to which 
the referrals would have changed. In order to do this, one 
must be careful that the alternative matches indicated by the 
system were, in fact, possible. This means that not only 
must the Jobs have been In open status at the time the client 
was available for referral, but that the alternative matches 
must also satisfy other criteria not measured by the system 
being examined. 

One can use this approach for any matching algorithm and, 
in fact, for examining what are the strategies now being 
used. One can, with this approach, also obtain the data 
needed to validate the alternative (and current) strategy, 
in a predictive sense, if the post-placement criterion meas- 
ures are obtained. In a sense» this is what Ultrasystems 
did in the SER experiment, except that we did not obtain 
the data needed to assess what the alternative matches were 
for a given client when the client was ready to be referred, 
nor did we obtain the data needed to assess the influence 
of other factors which also determine the alternatives really 
available. 

In addition, Ultrasystems believes it would be worthwhile 
to have the referral intennediary document in some manner 
the matches that were considered and the reasons why ones 
were rejected and/or chosen for referral. This should be 
done in both phases of the experiment. It would provide 
extremely valuable data, but it is not clear to us exactly 
how one would do this. 

4) Ultrasystems believes that the approach discussed above is 
the preferred way to Implement a comparative evaluation. 
The con^arison group should, in our opinion, be assembled 
prior to the experimental group. The CJMS instruments 
should be administered to the jobs and people who comprise 
the comparative group, but the scores should not be pro- 
vided. The implementation of an evaluation in this manner 
will provide data for use in several important analyses: 

a) It will provide data which can be used to assess what 
the difference is between the CJMS preferred (indicated) 
matches and the ones actually provided under existing 
processes. 
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b) It can provide data that can be used to assess the 
impact of alternative cutoff criteria, especially that 
which is associated with CAI. 

c) It can be utilized to test the use of procedures whereby 
job descriptions can be obtained which will enable one 
to assign the appropriate cluster with a given degree of 
confidence. Ultrasystems believes it may be extremely 
difficult for a manpower organization to consistently 
obtain JOCLs in time for their use in matching. In other 
words, if a large percentage of referrals and placements 
are made within a few days after receipt of the job order, 
then it is going to be very difficult, if not impossible, 
to obtain the corresponding JOCL in a short enough time 
period so that the time-to-fill performance is not ser- 
iously impaired. It Is Inconceivable that an organization 
would withhold a job opening because no JOCL was avail- 
able. Therefore, Ultrasystems believes It Is extremely 
Important that a proven methodology be developed which 
win enable an order taker to assign the appropriate 
cluster based on a verbal job description. 

Examination of recent performance data can reveal the 
extent of this issue. Repeated job orders from given 
employers can be profiled and stored for re-use. 

d) The generation of the comparison group prior to the 
experimental group will also enable one to determine 
the differential analyses that can be undertaken. In 
this regard, Ultrasystems believes it is important that 
one review recent data to assess the length of time that 
each of the experimental phases should last. One needs 
to examine the client and job characteristics obtained 
over varying time periods to determine how long it will 
take to achieve the desired cell sizes for the evaluation. 
Hopefully, the currently available data can provide the 
proper disaggregations so that one can determine the 
characteristics of the client and jobs that are within 
the range of applicability of the CJMS. One could also 
use the CJMS data obtained on the comparative group to 
assess the impact that cutoff criteria will have on the 
resultant cell sizes. 

5) Obviously, extreme care must be taken to Insure, to the 
degree feasible, comparability between the people and jobs 
in the two groups. The criteria associated with the use 
of the CJMS for both people and jobs must be applied to both 
groups. The examination of recent agency data mentioned 
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above can be used to determine the percent of agency clients 
and jobs which will meet the criteria as a function of time. 

Examining recent data can reveal the extent to which dif- 
ferential analyses based on client and job characteristics. 
Independently and pal red » can be undertaken based or agency 
volumes over time. In this regard, Ultrasystiyns believes It 
would be useful to see If one could assemble a special com- 
parative data base consisting of the same jobs at the same 
companies to which both client groups would have referrals 
and placements. This would enable one to con^are post-place- 
ment performance using the alternative match strategies hold- 
ing the job-company variable constant, tf one could also 
obtain frm these selected companies data on their overall 
retention in these job categories, one would have another 
valuable comparative measure. 

6) One could Ignore much of what has been said above and simply 
compare performance over given time periods where the system 
was not used and was used. One could do this without atten^t- 
Ing to determine to what extent the system was actually used 
on the premise that Its use could never be totally enforced 
or, stated another way, that Its use will vary depending on 
the judgment of the refenal personnel. In this sense, one 
would be Interested In determining why It was or was not 

used and might not really have comparative performance data 
reflecting Its use In all situations to the degree it was 
applicable. If the system Is truly valid in the types of 
situations that are a reasonable percent of the agency's 
operations, then one could base Its use on this validity 
without regard to Its overall effect on placement performance. 
One might still Implonent process changes to Improve job 
solicitation activities and to provide computer assisted bi- 
dlractlonal matching to Increase exposure based on the con- 
tention that this will Improve placement performance Inde- 
pendent of post-placement performance, 

7) The attalranent of CJMS profiles during the non-use portion 
of the experiment provides better data for use In predictive 
validation studies because of the wider range of Indices 
attainable. If cutoff criteria are used In the use phase, 
then this will tend to narrow the range of resultant scores 
for use In predictive validation against post-placement per- 
formance criteria. While there are statistical techniques 
available for dealing with restri<;t1ons In range, Ultrasystems 
believes It Is better to use the data obtained In the non-use 
portion for predictive validation studies. 
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DESCRIPTION OF THE CLEFF JOB MATCHINS SYSTEM 



This section presents a description of the design and operation 
of the Cleff Job Matching System. The details regarding the 
methodology associated with the development of the system, the 
basic tenets underlying the system, and the history of its de- 
velopment will not be discussed here. The reader is referred 
to Appendices A and B for this material. 

The CJMS is based on obtaining numerical scores frm applicants 
and about jobs for each of sixteen dimensions of work. These six- 
teen dimensions are divided into three categories: there are 8 
dimensions for Thing-oriented activities, 4 for People-oriented 
activities, and 4 for Idea-oriented activities. Each of the 16 
dimensions are defined in Table 3-1. The set of 16 numerical dimen- 
sions obtained is referred to as a profile. The s^une 16 dimensions 
are used on both the client profile and the job profile. 

The numerical scores for each of the 16 dimensions are obtained via 
what is referred to as a checklist. There are, in fact, two check- 
lists for use by clients and one checklist fOr use by employers in 
describing Jobs. In addition, there is a Card Sort System utilized 
in describing jobs. Each card pertains to one of the 16 dimensions. 
Hence, there are sixteen such cards. The content and administration 
of these checklists and the job cards will be discussed separately in 
the following sections. This will then be followed by a discussion 
of how the system matches or relates the clients' 16-dimension scores 
to the job scores. 

3.1 THE CLIENT SELF-INTERVIEW CHECKLIST (SICL) 

As stated previously, the nianerical scores for each of the sixteen 
dimensions for each client are obtained via Self-Interview Checklists. 
There are two of these checklists for use by each client. The client 
checklists are referred to as SICL's, which is an acronym for the term 
"Self-Interview Checklist." T4ie two SICL's are titled Likes and Dislikes 
and Done and Not Done. They are cowwnly referred to as the preference 
SICL and the experience SICL, respectively. The Likes and Dislikes SICL 
{the preference SICL) is used to obtain, for each dimension, nunerlcal 
scores that indicate what they, the client, like best and dislike most 
{what motivates them positively and negatively). The Done and Not Done 
SICL {the experience SICL) is used to obtain, for each dimension, numerical 
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TABLE 3-1. DEFINITION OF 16 CLEFF JOB-CLIENT 
MATCHING SYSTEM PROFILE DIMENSIONS 



T Thing-oriented Behaviors 

Job activities which immediately and directly involve 
the worker with things such that: 

T-1 Athletic 

- uses the large muscles of the body in hard 
physical labor 

- pays attention to broad overall results only 

T-2 Utility 

- helps others by doing unimportant things to save 
their time and energy 

- acts in response to some immediate demand by 
someone else 

T-3 Fine Manual 

- uses fingers in close coordination with the eyes 

- pays attention to details and small things 

T-4 Gross Manual—Independent 

- applies some skill in the use of hands, tools, 
or machinery 

- is relatively free from detailed supervision by 
procedure, supervisor, or machinery 

T-5 Gross Manual— Dependent 

- applies little skill in the use of hands, tools, 
or machinery 

- is dependent on detailed and close regulation by 
procedure, supervisor, machinery, or assembly line 

T-6 Order 

- puts things where they belong 

- cleans or tidies things 

- lubricates things 

T-7 Correction 

- corrects the quality in own work or in the work 
of others 

- insures quality performance of machinery and 
equipment 

T-8 Locomotion 

- drives or operates mobile vehicles 

- operates equipment mounted on vehicles 

- physically moves away from work space a good deal 
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TABLE 3-1 (continued) 



P — People-oriented Behaviors 

Job activities which inmedlately and directly involve the 
worker with people such that: 

P-1 Attendance 

- provides a service completely determined by 
the employer 

- has a very narrow range of decision 

- has minimal and superficial contact, 
never physical 

P-2 Physical Service 

- gives a rsgularized service which meets their 
personal needs or demands 

- is pennitted a relatively wide range of decision 
within specified duties 

- can have pl\ysical contact, but a superficial 
personal relationship 

P-3 Management 

- guides, influences, or directs the present and/or 
future ongoing behavior of others 

- takes or shares responsibility for results of that 
behavior, including their work 

- there is high potential for emotional relationships 

P-4 Persuasive 

- convinces and persuades others to react quickly in 
some way beneficial to employer 

- takes little or no responsibility for future behavior 
of other persons 
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I — Ideas aad Infomation-oriented Behaviors 

Job activities which iinnediately and directly involve the 
worker with ideas and symbols such that: 

I-l Verbal 

- uses written or spoken words to represent ideas, 
people, or things 

- describes situations and relationships, or solves 
problems, using words 

1-2 Numerical 

- uses numbers or number concepts to represent ideas, 
people, or things 

- describes situations and relationships, or solves 
problems, in numerical terms 

1-3 Clerical 

- records or orders data of any kind 

- processes paperwork intended to do that 

1-4 Innovative 

- gives a personal interpretation to a situation and 
finds an artistic, creative, or original solution 
to it or change In it 

- expresses some personal idea nonconventlonally 
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scores that indicate what they, the clients, have done most and 
done least. Depending on the client's work history, the experience 
SICL may or may not be utilised. In all cases, the preference SICL 
is utilized. 

Each of the two SICL's consists of fifteen pages with each page con- 
taining sixteen activity-oriented phrases or behavioral units. Figure 
^-1 shows a page taken from the preference SICL. On each page of 
each SICL, there is one activity-oriented phrase {or behavioral unit) 
for each of the sixteen dimensions. The Project Indianapolis report, 
shown in Appendix describes the methodology by which these phrases 
(and the sixteen dimensions) were developed. As described in that 
report, each phrase (or behavioral unit) consists of a set of words 
which describe a unique and meaningful activity and a behavior which 
has some objective. Each phrase consists of a verb and an object. 

After reading all sixteen phrases on a given page, the client then 
selects ten phrases. The rationale underlying the selection depends 
on which SICL is being completed. For example, with the preference 
SICL, the client selects from each page the two phrases which describe 
activities the client likes the very most and the two which describe 
activities the client likes the very least (or even hates). The client 
then goes through the remaining 12 phrases and selects the three which 
the client likes more than any of those remaining, and then selects the 
three which the client likes less than ar^y of those remaining. The 
client does this on each of the fifteen pages. The rationale underlying 
the experience SICL is the same except the term "likes" is replaced by 
the term "done," The instructions state that the client is to consider, 
in making the selections, those activities that the client has done and not 
done during the course of his/her lifetime up to the time that the checklist 
is being completed. The instructions further state that this is to include 
activities associated with not only jobs that the client may h^ve had, but 
also with those activities done "in school, play, at home— as a matter of 
fact, at any time." Appendix E contains the full set of instructions for 
administering the SICL's. 



3.2 THE JOB OUTLINE CHECKLIST (JOCL) AND THE JOB CARD SORT 

The Job Outline Checklist (JOCL) is similar to the SICL's described 
above. The JOCL consists of ten pages with each page again containing 
sixteen activity-oriented phrases or behavioral units. Figure 3-2 
shows one page from the JOCL. There are, in fact, two JOCL's, one for 
use with white-collar jobs and the other for use with blue-collar jobs. 
The difference between these two is in the wording of the phrases. Not 
all the phrases are worded differently. 



63 

1-51 



Circlo 



Circltt 

two 



1. INSPECT HOUSES . « (most) <leatt) 



2. BALE KXY OR PAPER <mott) (loft&t) 



3. OPERATE ROAD GRADER {mott) <loatt1 



4. COLLECT WEEKLY XHSURAHCE PAYMENTS 

(mott) (U*tt) 

5. FOLLOW COMPLICATED WRITTEN INSTRUCTIOKS. . . 
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13. MAKE OUT CLERICAL FORMS (most) (least) 



14. OPERATE PUNCH PRESS MACHINE.'. .. (most) (least) 



IS. GET VOTERS TO REGISTER (most). (Icrast) 



16. SORT LAUNDRY. 



Circle 
(more 



(more 
(more 

(more 

(more 

(more 

(more 
(more 

(more 
(mora 

(more 

(more 

(more 

(more 

(more 



Circle 
three 

(less) 



(less) 
(less) 

(less) 

(less) 
(less) 

(less) 

(less) 

(lass) 
(less) 

(less) 

(less) 

(less) 

(Ictis) 
(less) 



• (nost) (least) (more) (less) 

SICL 1971J4 

Figure 3-1. Sample of Page Taken From Preference SICL 
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JOCL-: 271- CW 

Figure 3-2. Sample of Page from JOCL 
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The JOCL Is to be completed by the job supervisor, or preferably by 
more than one supervisor, at a given employer, who each supervise 
employees working at jobs to be profiled. The mechanics of completing 
the JOa are the same as for the SICL, I.e., on each page (each page 
1s referred to as at group) ten phrases are to be selected. The In- 
structions for the selection of the phrases are as follows (the quota- 
tions are from the Instructions printed In each JOCl); 



1. Try to vistialize the ideal worker doing the 30b 
you are descHbing, 

2. Carefully rend all eitcteen activity '-^phrases in 
each group, 

3. Decide which two activities most resemble the 
behaviors most required by this 30b in order to 
do it best, 

4. Decide which two activities most resemble the 
behaviors least required (or most interfering) 
in this 30b. 

5. Of the remaining IZ activity '^phrases ^ pick the 
three which are most like what should be done 
on this 30b to do it right, 

$, Of the remaining 9 activity --phrases^ pick the 
three which are most like what is not required 
(or interferes) in the 30b. 



The instructions In the JOCL also state; 



Work quickly J don^t take too much time mulling over any 
one phraee or group of phrases^ your first coneidered 
impression is what is desired here. 

It helps to imagine that the sixteen activities in a group 
are all that are available to do the 30b you are describing^ 
and you mu$t pick the best and the worst for the 30b. 

After the job supervlsor(s) has completed each of the ten pages (groups) 
In the JOCL according to the above rules, the supervisor Is then asked 
to sort a deck of sixteen cards. There Is ons card for each of the 
sixteen dimensions of work. The front of each card contains a defini- 
tion of one of the dimensions. The definitions are essentially the 
same definitions as given In Table 3-1. The back of each card contains 
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examples of the activities that each dimension refers to. As an 
example, on the back of the T-1 Athletic dimension card, the fol- 
lowing examples are given: 

Examples: load truoksj carry heavy thirigsj carry 

large baakete of files^ push loaded carts ^ 
move furniture or equipment 

As With the JOCC booklet, there are two sets of these cards, one for 
use with white-collar jobs and the other for use with blue-collar jobs. 
The above example is from the wlilte-collar set. The examples for T-1 
Athletic, from the blue-collar set, are: 

Examples: dig ditches^ shovel coalj load trucks 

The job supervisor is asked to sort the deck of 16 cards according to 
the following rules {taken verbatim from the instructions In the OOCl): 

Sorting the card deck 

1. Bach card has a definition of a particular kind of 
work behavior on one side (in black) and examples 
of this kind of behavior on the back (in red). 

2. Read all sixteen definitions very carefully. 

3. Pick the five definitions whCch best describe what 
must be done an this job in order to do it right. 
Put them in a pile to the left. 

4t Pick the five definitions which best describe what 
must be avoided by a worker in this jobs or a* least 
required by the job* Put th&se five cards in a pile 
to the right. 

S. Go through each pile of cards to review what you kc^^e 
done^ Rearrange if necessary. 

$. Put the six cards not picked back in the envelope. 

7. Record your choices on the facing page and follow the 
instructions* 

8. Return cll cards to the envelope. 

Instruction 7 refers to the page In the JOCL booklet where the job super- 
visor lists the names of the five dimensions of work which were chosen 
that best describe what must be done on the job to do it right and the 
five dimensions which best describe what must be avoided by a worker in 
this job, or are least required by the job. (See instructions 3, 4^ and 
5 above.) The job supervisor then assigns a scale value to each of these 
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dimensions. For those five that are the best descriptions of worker 
activities required for doing the job as well as possible, the five 
scale values are defined as: 0ut8t<mding^ Very Great, Great, A Lot, 
Some. 

For the other five diwansions (least necessary to the successful doing 
of the job), the five scale values are defined as: Highly Interfering, 
Interfering, Umeoeeeary, Very Seldom Required, Unlikely. 

This completes the job outline portion of the Cleff Job Hatching System. 
Appendix E contains the full set of instructions for administering the 
JOCL, except for those which pertain to arriving at the numerical scores, 



3.3 ADMINISTRATION OF THE SICL ANO JOCL 

The preceding two sections have described the content of the SICL 
and JOCL instruments and the mechanics of their completion. This 
section will discuss the administration of the checklists. One 
aspect of the administration of both checklists is straightforward 
and deals with explaining the mechanics of completion. The in- 
structions given in Appendix E and the preceding discussion essen- 
tially cover this aspect. However, there is more to the adminis- 
tration of both checklists than just explaining the mechanics and 
then having the client or the job supervisor complete the check- 
lists by themselves according to the instructions. Essentially, 
however, the checklists are completed by the client or the supervisor 
on their own. 

Some of the other aspects of administration are, in Ultrasystems' 
opinion, not very clearly defined. The sections which follow will 
discuss these aspects of administration in terms of what Ultrasystems 
has Observed and has been told. The reader should note that Ultra- 
systiims was trained by personnel of AOP-POS, Inc. in the administra- 
tion of the checklists, observed checklists being administered, and 
discussed their administration with the staffs of the agencies 
involved in the New Jersey/Cleff experiment. 

3.3.1 Administration of the SICL 

One of the steps involved in SICL administration has to do with deter- 
mining the reading ability of the person who is going to complete the 
checklist. The Instructions prepared by the system's developers state: 



1-56 

• 68 



fh&ve are two ways of aolleoting expevienoe and 
preference data using the SICL. The first , and 
mat desirable mthodt (standard) ^ ^ 
otients describe their preferences and experiences 
by completing the SXCL th&neetves. 

The second method^ which is knom oe the "Oireat 
Sating Format (DBF) of the SXCL/' should only be 
used in those oases u^r« it is not possible to 
use the first method^ e*gf., the client is iltiter' 
ate, the client cannot read at the Bth'-grade level 
or above t etc. 

If the "reading level" of the client is unknown^ it 
is first necessary to determCne if he or she will 
aomprehsnd the words and phrases used in the SXCL. 
This Jnay be acaomplished by having the client read 
the "reading paragraph" (see attachment 2a) aloud to 
the counselor; if the client does ^ot make more than 
four (4) mistakes the SXCL may be i'self) admnisteved 
through use of the atandard method. Xf the client 
makes five (s) or more mistakes with the reading para" 
graph, the Direct Rating Format of the SXCL should be 
used. 

The reading paragraph referred to above and the Instructions for its 
use are shown in Figure 3-3. There is also a Spanish language vercton 
of the SICL. 

As discussed earlier, there are the mechanical aspects of administra- 
tion. In actual practice this includes reviewing the manner in which 
the client has completed one or more pages of the SICL, as they are 
completed, to make sure that the instructions are being followed. One 
should note that on any given page ten activity-phrases are to be 
selected in the order described earlier. One cannot select the same 
activity-phrase more than once, and one must select the correct number 
of phrases in each of the four categories {two most, two least, three 
more, three less). 

The other major area of achiinistradon has to do with the client's 
ability to understand or relate to the activity phrases given on each 
page of the SICL. The reading paragraph and/or reading lev^l require- 
ments discussed above are designed to insure that the person can read 
well enough to understand what the phrases mean and can do this in a 
reasonable length of time. However, there Is the possibility that a 
client may, after reading all the activity phrases on a given page, 
indicite that he or she cannot find any or enough phrases that he or 
she likes to do, or has done, or doesn't like to do, or hasn't done. 



ERIC 



1-57 



69 



ATOACHMBNt 2 a 

PERSONNEL DATA SYSTEMS, (NC. 

342 MAOISON AVimJI 
NtW YOnK. N.Y. 10O17 • (212) 6*7.1477 



REftPmS PARAGRAPH 

♦ 

There are many kinds of people, and many kinds of .jobs* 
Some people operate road graders, paint outside of 
houses, sell groceries in a store, or repaix automobiles* 
Other people may preach sermons in church, take neighbor- 
hood surtreys, demonstrate automobile parts for sale, or 
do research on historical events* And the jobs ask 
people to do various things; for example, feed children, , 
lay bricks, work with a microscope, interview job can- 
didates, and maybe even do mechanical drawing* There 
is no telling what a job might demand of a person. We 
know of a job where the worker ^^ras asked to supervise 
cleaning a building, read complicated instructions, read 
gauges on machinery, measure food Ingredients, and 
catalogue mail order merchandise* He found the job 
interesting because he was asked to do so many diffejrent 
things* 



Figure 3*3. Reading Paragraph and Instructions For Its Use* 
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PERSONNEL. DATA SYSTEMS, INC. 

342 MAOtSON AVKNUK 
NSW YO#IK.N.Y. toon ♦ 1312)MT'14TT 



CLEFF MATCHING SYSTEM 



Instructions to Counselor - Intervievfer 



Ask the candidate to read the readHsTG PARfiGRapa. 

Say, ^^We in this office are writing a brochure for our 
customers, the job applicants like you, and we want to 
make sure that we are using words and language which 
makes sense to you* We would like you to read it aloud, 
then tell us if you think we can make it better, and 
how to do that*" Or words to that effect. 

Permit the applicant to read it aloud without inter- 
ference, but help if he has trouble, in a sympathetic 
way of course, to get him through. 

Keep note of the number of words he cannot accurately 
read. Mispronouncing a word does not necessarily mean 
he cannot read it. Use your own judgment. 

If you are sure he cannot read five or more words in the 
paragraph, then do not refer him for the standard S.I.C.L. 
Instead, you may want to consider administering the Direct 
Rating Format of the S.I.C.L. to him orally. 

If you administer either the Standard or the Direct Rating 
Format of the S.I.C.L., introduce the applicant to the 
system - tell him about increasing the potential for a 
good job match. 



Figure 3*3. (continued) 



ERIC 



71 

N59 



a> 

a* 



o . 
a> 
a> 



a> «/> 

I — ^ «/> 

a> a> z> 

«/> c o 

o 

a> <M 

>> J= o 

a> I— c 

«/> 

*/> I — 
a> Q. <M 
«/> E I — 

O 

^ o 0> 

— XP 



a> o I— 



S o 

0> XP 
-COO* 
t/> t/> 

z> C9 a> 



>> 

0> XP 

a> 
a> c 

0> fO 



o 
o 
o 



3 <M O 
O ^ 

0> C «/> 



a> 

^ is 

XP CO 0> 

c ^ sz 



o 

3 o 2 



-M O 

1— a> 
o a. 

o o 
a> a> 



I 1— o sz 

>> O. C <M 

> 

C C XP 

O o 
C C 

O 3 >> c 
-C ^ o 

£ I— <— 
<M • a> 

0> O XP 

4rt a> 

0> O. C <M 



>>^ 

«/> C/> I — 
13 to 
O ^- 4^ 

> a> o 

-O XP 

o o 
a> - 

• -C .J 

<M <M O 
C 

O i/i^ 
a> 

1— a> -c 
o o 

<M 0> ^ 
O 

-M - -M 
«/>«/> (ti 
£ U 

to d5 -M 

t/l */> 
«) */> 

O */> 
Oi— 



<M I— -M C 
3 0> C -M 

I — ~ O. «/> 
0> O. 0> 
O -C rtj 

-M -M 
«/> fl> 
«J O-C > 
GO <M <M 

■M 

= <M O 
* 0> 0> (ti 

fl* «J 

*rt I— XP o 

T 2! S *rt 

c o ^ 0> 
O -M -M -M 

■M -M >> C 

rtj c a> 

■M O <M = 
Q> 

I/) I— ^ l/> 
O. «/> 



*o 0* 
0* -C 



0* 

0* 

O 




: I— O <M 

TJ O 
«/> <M 
* 0) 0* 
TJ -O -M l/> 
0* rtj 0* 
^ O O 
O <M 

0* XP ^ 
^ «/> C jCZ 



C/1 «/> 
-C 



o «/> a. 0* 

O. O <M 
D-XP O 
> O 

0* 0* 4^ «/> 



O 

.^i . 

XP «/> O 
0* ^ 1^ 

^ O 0* 

o o o 
«/> c 

0* V> 0* 

XP 

^ 01 
0* 

■M 0* 

0* O -C 
-C ^ +^ 

oo* 

<M «/> C 

C <M 
0* «/> 

C ^ O 
O f-^ 
o 0*t/> 

C 3 0* 

o c 

^ O.0* 
«J X 
1— 0* J- 
0* 

€ ^ 
«/>o 0* 



o 

<M 1 

XP I— 
0* O. 

<M 

O) 0* ^ 

XP 

0* 3 4^ 
«/> O 

o c ^ 
^ o 

o 

0* ^ 
-M 

-C o 

<M 3 

COO. 
O O- 

■M O 

u o 

■M l/> rtj 
«/> 01 

fe-*- o 

^ jC -O 

M- «/> 
O • (t) 

4^ 0* 
O t» 1^ 
O t/1 o 

l/> O rtj 
(ti «/> ^ 

^ XP «/> 
01 

■M */> 
O 3 E 

«/> C *J 
-r- <_> 

O 

0* 0.00 

u 

0* 0* 0* 

SZ JdSZ 
F- 



■M 0* 
C O 

O «) I — «/> <M O 
<M <M O. 3 O 

O. 0* «/> 
0* 0* O 1— ^ 
-C 0* 0* <s> 
_0 <M 0* -O «/> «/> 

«4- -M C 

Of O «) 

-O SZ 0* 

■M 3 <M >> «/> 

O ^ O <M 

■M -O XP 1* 

rtj O 3 -C 

«/>«/> > O O 

O C O M- 0* 
0* V> -C 

C -O 
O ^ O 

o g; 

E O D> O 
O */> C 
O. «/> I — 

* > C 
>> >> % 0* 

1— O 1^ -C 

3 i2 -C l/> o 

O -MO 

-O o */> c 

> -M e o 1— 

SOI <t) 
0* ^ 
C > «^ O 0* 
>V — 

• XP *rt 
^ 0*1— 
O 0* O ^ > 
f-^ ^ 4^ O 0* 
</> « — _Q -C 

0*- c 

x: -M xp c 

C O * 0* O 

O 0* 3 0* 

I — O l/> 

CO. «/> 

O O. O* O -O 

«J C -M TJ 

-MO «/> 

0> 0* >> C 0* 

^ _C «^-M O «/> 

O-M ^ -^-^ rtJ 

e o ^ -M 1* 

O 0* «J -C 

o c c/> _o 3 o 

«) -M 
«/> E 0* jC 

-M 0* -M • 3 

CM 0* 0* Q 

0* ^- • ^ g 

O O -M 0* 



I 

-M O 
«/> -M 

C 0* 

-M 

-i * 

- O 
*J -M 

-M 
CO C 
0* 

0* 

I— 
-M O • 
«/> 

O 0* 0* 
■M _C 4/» 

-M 

1* 

O 



0* 



<M 



_ 0* d* 

0* -P -M 

1— 0*1— 
o ^ 

0* 

O rtj -M 
0* -C 

-M 

0* 0* 
-M O 
C <*- O 

o 
>> «/> 
t/> 0* t/> 

■M -M O 

O <M -M 
«/> 0* 

^ o 

E «/> O. 

5 u 

-M C 0* 
*/> O -M 

*/> -M 



There is one other aspect of the SICL administration that deserves to 
be tnentioned. This concerns the decision that can be made as to 
whether or not the client is to be given the experience SICL. The 
rules governing this are not, to our knowledge, definitively defined. 
Our exposure has shown that this is a judgmental decision that is made 
based on the client's work history. For example, in actual practice, 
clients who are recent high school graduates and who have no work 
experience, may not be given the experience SICL. If they are given 
the experience SICL, the results obtained may not be used to the same 
degree as those obtained from the preference SICL. 



3.3.2 Administering the JOCL 

The administration of the JOCL is very similar to the SICL. Aside from 
the mechanics, the similarity includes the other issues just discussed. 
Ultrasystems' exposure to the system has shown that job supervisors also 
can have difficulties in completing the JOCL because of the literal 
meaning of the activity-phrases. Again, one is confronted with the situ- 
ation where the person completing the checklist indicates that they 
cannot select the proper number of activity-phrases on a given page. 
The situation is usually associated with the choosing of: 

1) the two phrases that most resemble the behaviors most 
required by this job in order to do it best, and 

2) the three phrases selected after the first four are 
selected (the two most and the two least) which are 
most like what should be done on the job to do it 
right. 

Again, the situation has to do with the literal meaning. The instruc- 
tions utilize the phrases "most resemble" and "most like" which indicate 
that the literal meaning is not what detemines the selection. In other 
words, the instructions do indicate thai: the job supervisor is to asso- 
ciate or interpret the literal activity-phrase with activities associated 
with the job in the sense that the activity-phrase resembles or is like 
these activities. The situation arises, however, where the job super- 
visor asks for help or asks for an example of how this is to be done. 
In practice^ it is common (or even required) for* the administrator 
to discuss the job in general tenns with the job supervisor, prioc to 
the supervisor's completion of the JOCL. It is also common to have the 
job supervisor show the JOCL administrator employees who are currently 
doing the job. In this way, the JOCL administrator obtains a "feeling" 
or an "understanding" of the activities associated with the job. This 
"feeling" or "understanding" of the job is usually transformed in the 
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JOCL administrator's mind into a view of the job activities in 
tenns of the sixteen dimensions of work. If the administrator 
knows (and the administrator should) the relationship between 
each activity-phrase and each dimension, it enables the adminis- 
trator to describe, in tenns specific to tiui job being profiled, 
the resemblance between a literal phrase anc^ the activities of 
the job. Thus the job supervisor gets a "feel" or an "under- 
standing" of what the tenns "resemble" or "most like" mean. When 
one is explaining the mechanics of the JOCL completion, one clearly 
indicates the tenns "resemble" and "most like." One does this not 
with examples, but in tenns of what the words resemble or most like 
mean. When a job supervisor indicates that he or she can't select 
enough phrases on a given page, one repeats the instructions using 
the tenns "resemble" or "most like." However, sometimes this is 
not enough and the administrator is faced with the decision of 
giving an example or of having an incomplete JOCL. 



3.'* SCORING AND MATCHING 

The Cleff Job Matching System uses the selected activity-phrases and 
the JOCL Card Sort to generate a set of sixteen numerical scores, i.e., 
one for each of the sixteen dimensions of work. These dimension scores 
are obtained by adding together numerical values that are assigned to 
each of the four categories into which an activity-phrase can be 
selected. The four categories are: Moet, Leasts More^ and Le$$. 
Activity-phrases not selected are assigned a value zero. The nianeri- 
cal values are positive and negative and the resultant set of sixteen 
dimension scores always adds to zero. 

The set of sixteen numerical dimension scores is referred to as a 
profile. Each SICL results in a profile; hence, there is an experi- 
ence and a preference profile. The JOCL and the Card Sort also result 
in two profiles. The profile obtained from the ten-page checklist is 
referred to as the Job Specific Profile. The profile resulting from 
the Card Sort is referred to as the Job General Profile. These two 
profiles are added to each other to obtain the job profile. 

For each client the experience and preference profiles are related to 
each other using two statistical measures as follows: 

1) the correlation coefficient between the experience and 
preference profiles is calculated and is referred to 
in this report as the Client Adjustment Index (CAI). 
It is also referred to as the Occupational Adjustment 
Index (OAI) 
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2) the sum of the squared differences over the sixteen 
dimensions is referred to in this report as the 
Client Difference Index (COI). 

In addition* a combined client profile is calculated. This ccxnbined 
profile consists of the average value of each dimension obtained from 
the experience and preference profiles. In those cases where the 
experience SICl is not administered* the combined profile is set 
equal to the preference profile. 

The systan uses the client and job profiles to obtain two numerical 
scores that indicate the "match" between a client and a job. These 
two nunerical scores are calculated and defined as follows: 

1) the correlation coefficient between the job profile and 
the client's conbined profile is calculated and is referred 
to in this report as the Job Match Index (JMI), or simply 
the Match Index (MI). It is also referred to as the Suita- 
bility Index (SI). 

2) the Sim of the squared differences over the sixteen dimen- 
sions between the job and the client's combined profile is 
calculated and referred to in this report as the Job Dif- 
ference Index (JOI)* or simply the Difference Index (DI). 



3.5 THE JOB CLUSTER REGISTER 

Over the course of time that the Cleff Job Matching System has been 
developed and used* hundreds of individual jobs have been profiled 
using the JOCL instrument. These job profiles have been assembled 
into a glossary that is called The Job Cluat&r Register. The glossary 
consists of individual job descriptions grouped into 19 clusters or 
general job groups. Each cluster is numbered but does not have a 
nane. Figure 3-4 shows the introductory page of the register and an 
exanple of one of the clusters. The cluster profiles were obtained 
by using cluster analysis. There are 525 unique jobs listed in the 
cluster register. 
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THE CJMS JOB CLUSTER REGISTER 

The CJMS Job Cluster Register is a glossary of job 
listings sorted into general groups of jobs or "clusters"* 
The computer print-out obtained from the Cleff Job Matching 
System provides the counselor with a match to a cluster 
number I and this register furnishes the descriptions of 
these clusters. Therefore, the register should be used 
only in conjunction with the Cleff Job Matching System to 
guide the individual in the selection of an occupation to 
meet his needs . 

Beneath each cluster profile in the register , a list- 
ing of various jobs appears. These jobs were collected 
and then sorted into the clusters such that the correlation 
between the specific job profile and the cluster profile 
would be on the average +.90, and not less than +.,80. 
This enables the counselor to suggest these jobs to the 
client as JT'iSsible selections with a high degree of confidence. 
However, the job titles listed under each cluster should not 
be interpreted specifically, but rather generally, since 
they are meant to refer to "types" of jobs rather than one 
particular job. When used this way, the effectiveness of 
the CJMS Job Cluster Register is maximized as a tool for 
both the counselor and the client. 

Figure 3-4* The CJMS Job Cluster Register 
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Cluster # 1 



(continued) 
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Cluster # 1 {continued) 
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Figure 3-4. (continued) 
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If one examines the sample cluster shown In Figure 3-4, one sees that 
each job is identified in terms of a job title, a subtitle, a business 
definition, and a six-digit DOT code. Not all the jobs listed have 
subtitles and/or business definitions and/or DOT codes. Of the 525 
jobs listed, 235 have a six-digit DOT codf^. The number of jobs listed 
under each cluster varies from 1 to 78. Neglecting the cluster with 
only one job title, the average number of jobs listed under each cluster 
is 29.1. 

The cluster profiles provide a mechanism for matching clients to jobs 
in the absence of, or in addition to, actual job profiles obtained using 
the JDCL. 



3.6 EXAMPLE OF A CJMS OUTPUT ^ 

Figures 3-5 and 3-6 show samples of CJMS printouts. Figure 3-5 shows 
an example of a client who completed both checklists matched to the 
Cluster Register. Figure 3-6 shows an example of a client who only 
completed the preference SICL matched to JOCL's. Both figures show 
an example of one approach to the presentation of the CJMS outputs. 
The figures show the following: 

» The client's SICL dimension scores are Dresented both 
graphically (in the fonn of a histogram) and numerically. 
The histogram shows both the preference and experience 
scores for each dimension, using the letters P and E, 
respectively. The number of P's and E's printed corresponds 
exactly with the client's actual numerical dimension scores. 

« The client's Occupational Adjustment and Difference Indices 
are shown. These indices only appear when the client completed 
both checklists. The example case, shown in Figure 3-5, shows 
that the client had an Occupational Adjustment Index of -41 
and a Difference Index of 1872. The Occupational Adjustment 
Index as presented is 100 times the correlation coefficient 
value, i.e., 100 times -.41. Figure 3-6 shows that for a case 
where only the preference SICL was completed, these indices 
values are labeled NA for Not Applicable. 

• The client's experience, preference, and combined dimension 
scores are presented numerically. 
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Figure 3-5,* CJHS Printout: Example of Client Who Completed Both 
Checklists Matched to the Cluster Register 
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Figure 3-6, CJMS Printout: Example of Client Who Completed 
Only Preference SICL Matched to JOCLs 
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3.7 



IMTERPRETATION OF THE MATCH AND CLIENT SCORES 



The discussion to this point has briefly described what the CJMS 
checklists look like; how' they are completed; and how the system 
generates numerical scores relating client profiles to job or 
cluster profiles. The subject of this section is the interpreta- 
tion of the profiles and profile-matching scores generated by the 
CJMS. 

Appenci.x F presents a document that addresses, in detail, the inter- 
pretation of the system scores and their use in vocational guidance 
and placement. Ultrasystems strongly urges the reader to read this 
document. The document mainly discusses the interpretation of the 
individual client profiles {preference and experience) and the 
interpretation of the comparison of these profiles. The document 
has been written for use by vocational guidance counselors and, 
specifically, for use by such counselors who are woHting with physi- 
cally or emotionally incapacitated (handicapped) people. 

The document specifically scates the following "rules of thumb": 



; , In using the CJMS ... any ... Oooupatiaml Adjustment 
Index aoore helow +.25 ia a strong indioation that the 
client may indeed be ooaupationatty maladjusted <md is 
in need of special attention in the 30b matching Prooees^ 
as well as oouns&ling help. (Hote: The Occupational 
Adjustment Index is the correlation coefficient obtained 
from the client's preference and experience profiles. It 
is usually referred to in this report as the Client Adjust- 
ment Index.) 

2. A dient with a Ion Prefermoe to ExpeHeme index (Occupa- 
bional, or Client Adjustment Index) should not be vefewed 
into a gob or typii of gob unless his Person to Jab Match 
IndAx (Job Match or Suitability Index) is over -h.eo. TJhis 
makes tt even more certain that both (the) preference and 
experience (of the person) has some positive relationshtp 
to the requirements of the gob^ thus giving {the person) a 
chance to succeed at the 30b cmd to reaover seme personal 
stability. 
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3. A client with a very high Preference to Experience Index 
con safely be referred to a job where the Person to Job 
Match Index is aa lew aa +.40^ because (the person) will 
bring personal stability to the job situation. 

4. The correlations called the Match Index (i.e.^ Job Match 
or Suitability Index) indicate to what degree the shapes 
of the profiles (i.e.^ clients combined profile and the 
job or cluster profile) are alike. The Difference Index 
(Job Difference Index) indicates the extent of the dCf' 
ferences between profiles and helps to break ties between 
matches, ^or example^ if a client's profile matches a 
nmber of jobs at approximately the sme level of itatch 
Indext i^e one with the smallest Difference Index would 
he the best match. 

The document in Appendix F also contains a table that relates the Job 
Match Index to the Job Difference Index. This table shows, for a 
given Job Match Index (between +.25 and +.94) what the median Dif- 
ference Index is and what the normal range of the Difference Index 
is. The description accwnpanying the table states that the normal 
range indicates the expected distribution of the Difference Index 
for a specified Katch Index, and further states that if the appli- 
cant's Difference Index exceeds the range for a given Match Index, 
there is iikely to a highly significant difference in one or 
more dimension comparisons and that this serves as a "red flag" to 
the interviewer. The description also states that the lower the 
Difference Index for a given Match Index iSs the less the likeli- 
hood is that there is a major dimension difference between the 
applicant's profile and the job profile, 

The use of Cleff Job Matching system for either matching a client 
to a job or for counseling a client in terms of vocational choice, 
involves the analysis of: 

1) the individual client's profile dimension scores 

2) the relationship between the client's preference and 
experience profiles 

3) the Job Match Index and the Job Difference Index between 
the client's combined profile and the job or cluster 
profile 

4) the relationship between the Job Match Index and the 
Job Difference Index 
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5) the relationship between the Job Match Index and 
the Occupational (Client) '.djustment Index 



6) the relationship between the client's profile 
dimension scores and the job profile dimension 
scores. 

The system is based on the profiles in terms of similarity of shape 
and magnitude of divergencies. The indices provide mathematical 
measures of the similarities and the divergencies. Examination of 
the profiles by inspection is used to determine divergencies and 
similarities in terms o." individual dimension scores. In the most 
simplistic sense, the CJHS indicates the relative appropriateness 
of client-job matches by the degree to which the client and job 
profiles have similar shapes and dimensional magnitudes. Again, in 
the most simplistic sense, the higher the similarity and the smaller 
the divergence, the better the match. Taken to the limit, this 
simplistic interpretation would indicate that a Job Match Index of 
-M.Q and a Job Difference Index of zero represents a "perfect match." 
If this were acconpanied by a ClienTWjustment Index of +1.0 and a 
Client Difference Index of zero , one would also have a situation where 
a person's experiences and preferences are "perfectly matched." 

The article shown in Appendix A and the report shown in Appendix B 
provide sane further insights into the interpretation of the CJMS 
scores. Ultrasystons urges the reader to read these documents. In 
regard to interpretation, the documents provide data which shows the 
means and standard deviations of the syston's scores associated with 
different definitions of good and bad matches. These data were 
obtained as part of the system's development. 

There are two other points that Ultrasystems believes are worth 
mentioning before this section is concluded. These two points 
relate to the following subject: 

1) the interpretation given to very high job match and 
Client (Occupational] Adjustment Indices- 

2) the specific job situation interpretation of the 
CJMS scores. 

The first point refers to instances in which the Job Match Index and/or 
the Client Adjustment Index is, say, equal to or greater than +.8. 
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was then followed by another experimental period where the CJMS 
would be used, again for approximately 1,200 clients. The next 
phase was to be a non-CJMS phase involving 200 clients, which was 
then followed by the use of the CJMS for the final 1,200 clients. 
In other words, there were to be three experimental phases, each 
of which was to Involve one-third of the total group of 3,500 
clients, and three control phases, two of which Involved 200 
clients each, and one of which Involved 400 clients. The basic 
purpose of these phases was to enable one to analyze the resultant 
data to see If there was a beneficial spin-off associated with the 
use of the CJMS on the traditional agency placement process. This 
was referred to In the experimental design as a "Hawthorne Effect." 
In addition, the phasing of the experiment would enable one to 
determine If the results obtained with the use of the CJMS improved 
as experience with Its use Increased. 

In addition to the phasing of the experiment, there were other design 
ground rules established as follows: 

• Provide for a distribution of Black, Anglo, and 
Spanish-speaking persons in the experimental group 
that is consistent with the representation of each 

roup among the unemployed and underemployed. 
Note: The wording is taken from the Statement of 
Work established for the experiment,) 

• Provide for a control group of persons proportionately 
representative of the experimental group. In other 
words, the control group clients were to have the same 
demographic characteristics distribution as the clients 
in the experimental group, 

• Each of the three experimental groups was to consist of 
every agency applicant meeting both agency criteria and 
CJMS criteria for placement consideration, until one-third 
of each agency's quota was reached. 

• The control group clients were to meet the same agency 
criteria for acceptance as the experimental group, 

• The experimental group clients were to consist of all 
Individuals processed through CJMS by each participating 
agency and who were referred either to jobs or to formal 
training on the basis of their match to jobs or training 
situations. An applicant will be referred only If the 
applicant shows a Suitability Index (Match Index) of 50 or 
higher to that situation, and has an Occupational Adjustment 
Index (Client Adjustment Index) greater than 25. 
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• Provide for referrals for both the experimental group 
and the control group to public and private sector 
jobs at a ratio of 1 to 2. 

• Provide for recruitment, matching and job profiling 
through (1) the Business and Industrial Coordinating 
Council (8ICC) of Newark, New Jersey, (2) Kodel Cities 
projects and (3) the New Jersey Rehabilitation Commis- 
sion in groups of approximately 1000-1200 applicants for 
each of the above three organizations. The final refer- 
ral and job placemen' to be the responsibility of each 
of 'the above organize \ns in accordance with their 
regular operating procedures. 

The experiment was to obtain and provide the data generated and 
needed for the evaluation. All the data needed for the evaluation 
was not the responsibility of the experiutental contractor. Specifi- 
cally, the capture of the post-placement data needed for the evaluation 
was the responsibility of the evaluatio 'tontractor. 

As stated above, the basic objective of the evaluation was to determine 
whether the CJMS is superior to traditional placement mtthods. Uhereas 
the Request For Proposals for the evaluation did not explicitly state 
what the term "traditional placement methods'* meant, it did describe, 
in general terms, the Diatiojica^y of Oaoupatioml fitlea. The RFP 
stated the following points relative to the Hetioncay of Ocanpational 
Titles (DOT): 

m 

• !the DOT wa3 tha firat 8y8tem<;tHc attmpt to get at tha 
prcblm of impi^oved Job ptacmmt. 

• Tne DC^ is a classification scheme which aimmges ocrupa-- 
tiom according to their interrelationships^ standardizes 
Job titles^ and defines^ generally^ what each Job involves^ 

• As a tool for identifying occupational areae and the 
general Hnds of Job titles and Jobs which ocaur^ the 
DOT is useful. 

The RFP went on to state that: 

• The kinds of inform^Hion available in the DOT appear to 
be of little value in attempting to match on individual 
person to an individual Job at entry or semi-skilled 
level. An example of this rigidity is that identical 
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DOT codes assume idmtical 30b charaaterisHcs when^ 
in fact^ no two dobs are identical. Jjikeufise^ no two 
individuals ax^ identical^ yet the assignment of th^ 
same DOT code to two individuals assumes this mivoaal 
relationship. 

• Ma^ov weakness of other 8yste?ns is that they require 
that people be described on the basis of job eTsgerienae. 
This approach is highly questionable from several 
standpoints when ueed for matching at the entry and 
semi-skilled level. 

1} If an individual does not have any 30b experience^ 
no match can be made. 

2) It assmes that an individual that does have job 
experience is only suitable for a Job in the area 
of his experience. 

3} The primary mechanism for detemining an individual *s 
experience^ suitability and likes and dislikes is the 
subjective Judgment of a counselor/interviewer. 

4) Job Analysis tends to be based primarily on employer 
opinion of requirements rather than on scientific 
assessment of Job content. 

The RFP then briefly described what was the rationale behind the 
experiment and Its subsequent evaluation* 1.e.» to explore poten- 
tially more effective approaches to matching disadvantaged Indivi* 
duals and jobs and to address the above-stated problems. The 
experiment was to provide the data and the subsequent evaluation 
was to determine If the CJMS was superior to traditional job place- 
ment methods » based primarily on the comparison of subsequent job 
retention rates, job satisfaction, and job performance between the 
control and experimental groups. The traditional job placement 
methods would be whatever the participating agencies utilized prior 
to and between the use of the CJMS. In addition, other success 
criteria was also to be consifdered, such as placement rates and 
employer satisfaction with the system. 

The New Jersey/Cleff experiment and Its subsequent evaluation was 
therefore to be primarily a comparative analysis of the placement 
and post-placement results of the two client groups. Theoretically, 
at le?s;t, all, except the use and nonuse of the CJMS, between these 
two groups was to be assumed equal. 
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The research design established for the evaluation was essentially 
straightforward and traditional. Since the experiment embodied 
within its design the creation of the experimental and control 
groups, there was r^o sample selection required for the evaluation, 
i.e., the samples had already been chosen. In addition, the data 
needed to describe the socioeconomic characteristics of the clients, 
the referral and plac^nent activities that took place and, for the 
experimental group, the CJMS profiles and scores associated with 
these clients and their referrals/placements, was to be provided 
by the contractor running the experiment. These data would be uti- 
lized to determine the following: 

1) Statistical comparisons of the demographic characteristics 
of the experimental and control groups. 

2) 'Statistical comparisons of the referral and placement 

activities of the experimental and control groups. This 
area of analysis Includes such measures as: 

i, percent of clients referred to jobs and/or 
training 

ii. percent of clients referred who were of^'ered 
jobs and/or accepted for enrollment in training 

ili. percent of clients referred who were placed into 
jobs and/or training 

iv. percent of clients who accepted job offer/training 

V. percent of clients who failed to report to work 
after accepting job or who failed to report to 
training after accepting 

vi. percent of total clients who were offered jobs/ 
training 

vii, percent of total clients who were placed/enrolled 
in training 

3) Statistical analysis of the referral and placement results 
of the experimental group clients as a function of the CJKS 
scores. This area of analysis includes such measures as: 

i, percent of referrals offered a job/training 

ii. percent of referrals offered a job/training 
who accept 

iii. percent of referrals who report to work/training 
after accepting a. job/training offer 
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As previously Indicated, the New Jersey/Cleff experiment failed 
to capture sufficient data for the evaluation to be conducted 
according to the original statement of work and, hence, in 
accordance with the proposed research design. Ultrasystans 
did obtain and analyze data related to the referral and place- 
ment activities associated with the New Jersey/Cleff expenment 
and did conduct interviews with the staffs of the agencies 
participating in the experiment. No interviews were conducted 
by Ultrasystems with any of the clients placed by the agencies 
during the experiment; with any on-the-job supervisors; or with 
any personnel officers. The results of the activities completed 
by Ultrasystems, regarding the New Jersey/Cleff experiment, are 
presented in Part II, Sections 2 and 3. It should be pointed out 
that the data obtained from the New Jersey experiment were not 
sufficient for the post-placement analysis, i.e., items 4 and 5 
above, nor for the referral/placentent/CJMS score analysis (of the 
experimental group) as outlined in item 3 above. The contractor, 
ADP-PDS Inc., who conducted the experiment, submitted a final 
report to OEO in June 1973. This report contained data and analy- 
ses of the referral/placement activities of the experiment. In 
addition, their report presented an analysis of the comparative 
retention of a small sample of experimental and control group 
placements and of the retention of the experimental group hires. 
The analyses done by ADP-PDS, as presented in the final report, will 

be reviewed in Part II, Section 3. The data utilized by ADP-PDS in 
the comparative retention analysis were not supplied Ultrasystems. 

The problems associated with the conduct of the experiment, in light 
of the requirements for the evaluation, were documented by Ultrasystems 
in a technical note submitted to OEO in August 1972. This technical 
note was based on Ultrasystems' initial visits to the agencies parti- 
cipating in the experiment. The purpose of these visits was to 
ascertain the status of the experiment, the procedures utilized in 
the experiment, and the availability and content of the data to be 
supplied. The technical note raised some serious questions and formed 
the basis for a series of discussions between OEO, Ultrasystems, and 
ADP-PDS, Inc. It should be noted that by the end of August 1972 the 
experiment was almost over. Therefore, whatever had happened regarding 
the manner in which the experiment had been conducted, could not be 
undone. The one area that was both crucial and apparently fixabie was 
the data base that was needed to do the evaluation. Ultrasystems 
.^veloped a data collection form that was to be utilized to record 
the data needed for the evaluation. This data form was to be used 
by the experiment's contractor to record the data needed to do mairy 
of the evaluation analyses and to enable Ultrasystems to locate the 
clients and supervisors to be interviewed. The technical note did. 
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however, present what were* in Ultrasystems* opinion, some serious 
questions about the manner in which the experiment had been con- 
ducted. The most serious of these questions was concerned with 
the relationship between the CJMS client-job match scores and the 
actual job{s) or training that a client was referred to and, 
possibly, placed in. In order to do the analyses involving the 
CJMS match scores and the referral/placement outcomes and the post- 
placement outcomes, one needs to be able to match the client with 
the job and/or training program. The matching can be done using the 
JOCUproduced profile or the appropriate cluster profile. The actual 
Implementation of the Hew Jersey/Cleff experiment did not result In a 
JOCL profile being obtained for every job opening and/or training 
program associated with the experimental referral groups and/or place- 
ments. It also did not appear that the job descriptions available from 
the data base would all be sufficient to enable one to assign each of 
them to a t;luster. 

In addition, the comparative experimental/control group analyses depend 
heavily on the differentiation between the two groups associated with 
the use and nonuse of the CJMS* In other words, it was, and still is, 
Ultrasystems' opinion that the cause-effect relationship between the 
referral/placement/post-placement outcomes and the use of the CJMS 
could not be inferred unles? the resultant CJMS client and match indices 
were utilised to direct the referral process. The Inferences to be 
drawn between the experimental and control group outcomes could not be 
solely due to the event of completing the SICL's and executing the CJMS 
match algorithm. The experiment had a specific ground rule that stated 
that a person would not be referred to a job and/or a training program 
unless the match index was greater than h*.50 and the person's Occupa* 
tional Adjustment Index was greater than +.25 {or maybe +.20, as indi- 
cated in ADP-PDS's first progress report). Ultrasystems, therefore, 
was concerned that unless the match indices could be definitively 
ascertained for each referral and placement, the value of the com- 
parative analyses would be compromised. 

There were some other problems associated with the experiment. Two of 
the more important ones were: 

1) The fact that the control group for the agency that had, by 
far, the largest experimental group was composed of clients 
of another agency not participating In the experiment; I.e., 
the agency was BICC which accounted for 60% of the total 
experimental group of 2619 clients. * 
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2) The fact that in all the participating agencies, except 
8ICC, the CJMS was not given to every client in the 
timespan when the system was being used. The experimental 
design had stated that the experiment groups "will consist 
of every agency applicant meeting both agency criteria and 
CJMS criteria for placement consideration until. ..the par- 
ticular agency quota had been set." The experimental design 
went on to state - 

These experimental eubo^eats are all those individmla 
proaess&i through the CJMS by each participating agency 
(md are referred either to jobs or to fomal training 
on the basis of their matoh to job or training situation. 
An ccpplioant will be referred only if he sha^s a Suita^ 
Hlity Index of SO or higher to that situation* <md has 
an Oaaut^ational Adjustment Baator greater than 2S. 

These statements do not imply that every person coming to one of the 
participating agencies is to be given the SICL's during each of the 
three experimental periods. In fact, the statements are not really 
very clear. Th^ initial statement does clearly state that the person 
must meet the agencies ' crit eria. _Therefore, ijt^ iiiplies that each 
experimental group client will 'have'bee'rr'acceptecrby the agency and 
that the agency will undertake activities on behalf of each client, 
especially as regards job and/or training referrals. The statement 
that indicates the client is also meet CJMS criteria for placanent, 
is not so clear. What are the CJMS's criteria for placement? As stated 
by the system's developer, bhe CJMS has been developed to provide a 
means by which jobs and job applicants can be matched at the semi- and 
low-skilled levels. The completion of the SICL in written form requires 
a certain reading ability. However, the SICL can be administered orally 
when this condition is not satisfied. The cut-off criteria on Match 
Index and Occupational Adjustment Index require that the client complete 
both SICL's and that the client be matched against, at least, the cluster 
register. Thus the client has to take the SICL before these criteria 
can be applied. In fact, the experiment was conducted such that all 
clients who completed one or both SICL's were included as experimental 
group clients, regardless of their Occupational Adjustment Index. In a 
addition, in many of the agencies {excluding BICC and TOPS), it is a 
fact that not every client who was accepted by the agency during the 
experimental phases took a SICL. In fact, there were no experimental 
phases, as originally set forth, and Ultrasystems has never seen the 
quotas that were to be established for each agency. 
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It should be noted, however, that data to be obtained by Ultra- 
systems through the interviews to be conducted with clients and 
on-the-job supervisors could have been used to resolve several 
of these problems. Most importantly, the interviews could have 
provided a means for definitively ascertaining the client's Job 
Match Indices, either in terms of a cluster identification, or 
through the administration of a JOCL. However, In December of 
1972, OEO decided that it was not cost effective to proceed with 
the client and the on-the-job supervisor interviews. Instead, 
OEO proposed a new evaluation scheme and directed Ultrasystems 
to redesign the evaluation, using their proposed scheme as a 
guide. Ultrasystems submitted a technical note in January 1973, 
In response to this request. In addition, Ultrasystems proposed 
that a new mini-experiment be conducted under the direction of 
Ultrasystems. This proposal was accepted. The so-called mini- 
experiment was to be conducted and evaluated by Ultrasystems. 
The experiment would be run In conjunction with four California 
offices of Project SER. This experiment and its evaluation will 
be described in the following section. 



4.2 SCOPE OF THE SER/CLEFF EXPERIMENT ANO EVALUATION 

The basic objective of the SER/Cleff experiment was to obtain reten- 
tion data on a set of 150 clients who were placed into jobs through 
the services provided by four offices of Project SfeR. Each of the 
ISO clients would have completed one or both SICL's, and each job 
that the client was placed into would be JOCL'd. The minimum re^ 
tention span would bs greater than six months,, i.e., each client's 
retention would be tracked for a minimum of six months after hire, 
or until termination, whichever came first. The basic objective of ' 
this experiment was, therefore, to obtain sufficient data to analyze 
the relationshi[i between the CJMS client-job match scores and the 
resultant retention. The experiment and its evaluation did not call 
for interviews with those clients who were placed and their on-the- 
job supervisors, fo.- the purpose of obtaining post-placement measures 
of job performance and satisfaction. In addition, the experiment did 
not include a control group. 

The implementation of the SER/Cleff experiment resulted in the tracking 
of 142 placements for a minimum of 14 months. 
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The new experiment and evaluation differed substantially from the 
New Jersey experiment and evaluation in its basic design and 
philosophy. For one, no cut-off criteria were established for 
the SER experiment. In other words, no minimum values of the 
CJMS indices were established to govern the referral process. 
In addition, the job profiles would be obtained after the fact, 
i.e., after the client had been placed. 

Thus, the SER/Cleff experiment was aimed at providing data to enable 
one to analyze the ability of the. CJMS to predict subsequent job 
retention. It was not designed to test the hypothesis that the CJMS 
is superior to traditional job placement methods. The New Jersey/ 
Cleff experiment was based, in Ultrasystems' opinion, at least 
implicitly, on the fact that the CJMS does, if utilized according 
to its design, provide measures that predict, in a relative sense, 
the level of retention, satisfaction and performance of job-client 
matches. The New Jersey/Cleff experiment and evaluation was, then, 
to test the hypothesis that the use of the CJMS, according to its 
design, provides manpower personnel with decision-making information 
that they do not ordinarily possess or use. The SER/Cleff experi- 
ment was, in a sense, going back to square one in that it was to 
determine if a relationship does, in fact, exist between the system's 
match and client measures and the eventual retention of job placements. 

The SER/Cleff experiment would, because it established no CJMS index 
criteria for referrals, enable one to obtain data on the matches that 
manpower personnel do, in fact, make, viewed from the standpoint of 
the CJMS measures. However, as was pointed out in the conclusions 
section^ one cannot infer from this that these are the matches that 
the manpower personnel considered to be "the best." There are many 
other factors associated with client-job matching that were not measured. 
One of the most important of these is the availability of job openings 
at the time the client was to be referred. In other words, the SER/ 
Cleff experiment did not obtain data that would enable one to know 
the alternatives that were available at the time the client was referred. 
Thus, matches could have been severely constrained by the availability 
of job openings and the client's need for a job. 

The design and execution of the SER/Cleff experiment and its 
evaluation both gained and suffered because of Ultrasystems' 
exposure to and analysis of the New Jersey/Cleff experiment. 
The conclusions reached by Ultrasystems include several associ- 
ated with the experimental and evaluative methodologies that, 
in our opinion, would work best for ascertaining the effectiveness 
and usefulness of the Cleff client-job matching system and, in 
fact, other matching systems. These conclusions are the benefit of 
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hindsight. As stated earlier, the SER/Cleff experiment benefited 
and suffered from the lessons we believe we had learned from the 
Hew Oers^ experiment. The lessons learned from the New Jersey 
experiment manifested themselves in the design of the SER/Cleff 
experiment as follows: 

1) Ultrasystems was convinced that the administration of 
SICL's and the capture of the required data could best 
be done if it was made the sole responsibility of a 
given person at each office participating in the ex- 
periment. Therefore, the experiment provided for such 
an individual in each of the four SER offices partici- 
pating in this experiment. These people were employed 
by Ultrasystems but were selected by the SER offices, 
based on a job description and requirements set forth 
by Ultrasystems. 

2) UUrasystems was convinced that brief, one-shot, or even 
a few repeated training sessions, were not adequate. 
Therefore, the SER experimental methodology included, 

in addition to an initial training session at each SER 
office, the hiring of one of the senior counselors from 
8ICC to provide continuous instruction on the use of the 
system for the entire period during which the SICL's 
were being adninistered. This individual, Mr. Ron Harris, 
also provided training on the adninistration of the JDCL's. 
Mr. Harris had been involved with the use of the CJMS at 
8ICC for approximately two years prior to his joining Ultra- 
systems for work on this experiment. He had personally 
administered the major portion of the JCCL's done by 8ICC. 
His qualifications were heartily endorsed by the senior 
personnel at ADP-PDS. The initial training was given to 
the SER office staffs by Mr. Michael Youchah of ADP-PDS. 

3) The operation of the New Jersey/Cleff experiment convinced 
Ultrasystems of the necessity of maintaining an independent 
set of client files to provide the data necessary for the 
evaluation. Df paramount importance was the capture of the 
data that definitively described the job that a client was 
placed into. Since the SER/Cleff experiment called for the 
JDCL's to be administered after the fact, it was imperative 
that data be captured that provided Ultrasystems with the 
job title, employer name and address, and supervisor name 
for the SER clients hired. The people employed by Ultra- 
systems, who worked in each SER office, were, in addition 
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were interested in exploring the use of the CJMS as a 
tool to assist them in their operations. Ultrasystetns 
explicitly stated, to the SER personnel, that we could 
not and would not vouch for the usefulness and effective- 
ness of the CJMS. Thus, we made no claims as to the 
system's validity. Instead, we tried to make it clear 
that the purpose of the evaluation was to provide some 
measure of the systan's effectiveness. The experiment 
operated in a shadow area between actual use according 
to the system's design, and no use, {i.e., operations 
solely devoted to the collection of SICL scores, JOCL 
scores, and retention data). 

It is essential to point out, that it is Ultrasystems' opinion, that 
this lack of clarity regarding the true purpose and execution of the 
experiment has no effect on the analyses done with the data obtained. 
The SICL's were administed by trained personnel according to the sys- 
tem's design. The JOCLs were administered, after tlie placement, by 
Ultrasystems' personnel trained in their achiini strati on. The reten- 
tion of the individuals was obtained directly from the employers, by 
Ultrasystems. The three essential ingredients for the analyses were 
not compromised in any way by the use made of the system by the SER 
counselors. 

There were some other aspects of the SER/Cleff experiment that deserve 
mention in the context. of lessons not learned from the New Jersey ex- 
perircant, and/or lessons learned from the SER experiment. These are 
as follows: 

1) The retention follow-up data was obtained through tele- 
phone contact with those employers who hired clients and 
who completed the corresponding job JOCL. These telephone 
contacts were made at four points in time. At the first time 
point, i.e., Decenber 31, 1973, all the enployers who had 
hired clients and had completed the corresponding job pro- 
file were contacted. At each subsequent time point, only 
those employers who reported that one or more clients were 
still working at the preceding time point, were contacted. 
Once an anployer indicated that the client or clients hired 
by than had all terminated, they were no longer contacted. 
It turned out that on at least two occasions employers who 
had previous.ly Indicated that the client was still working, 
as of the preceding contact time point, now indicated the 
client had terminated prior to that time point. This was 
due, they said, to a time de.lay associated with the processing 
of their termination records. This is a general type problem 
associated with retention tracking, and has, obviously, no 
relationship to the CJMS. It can lead, however, to the incor- 
rect recording of subsequent retentions. 
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2) In contacting employers to obtain retention follow-up 
data, Ultrasystems also asked for the reason for the 
termination. There were many such reasons obtained. 
Some of these are not that definitive, i.e., quit, no 

' reason given; went on personal leave— never returned; 
personal reasons. The purpose behind this was to pro- 
vide data that could be utilized to determine if the 
termination was related to the measures of the CJMS 
or was due to .some other reason not measured by the 
CJMS. As an example, terminations due to pregnancy, 
due to lack of driver's license, or due to falsifica- 
tion of a birth certificate, are, in our opinion, 
terminations not associated with the characteristics 
measured by the CJMS. Because it is difficult, if not 
impossible, to consistently obtain termination reasons 
that are clear enough to decide whether they are asso- 
ciated directly with the measures provided by the CJMS, 
it becomes difficult to make these distinctions in the 
analyses. In addition, Ultrasystems is aware, based on 
readings of other research/evaluation activities, that 
one can obtain different answers from employers and 
employees about the same event. There is a difference 
in perspective between the two. Ultrasystems has no way 
to know the extent of these differences for this evalua- 
tion, as clients were not contacted regarding the reasons 
for termination. 

3) In tracking retention of clients emnloyed at a given compaoy 
one IS faced with a decision related to determining^ if still 
employed, what job they are employed in, and when did a job 
change or changes occur (if they did). In doing this evalua 
tion, Ultrasystems did not grasp the significance of this. 
Therefore, our retention was done in terms of working or 
terminating from the company, and did not explicitly deter- 
mine if job changes within the company had occurred. This 
issue is considered, by Ultrasystems, to be of considerable 
significance, since the CJMS is matching a given client to 
given jobs, i.e., not to the company. The movement of a 
person to a "better" job within a company could be, and 
probably is, a good measure of the quality of the initial 
match. However, one cannot assiane that job changes are 
always upward or to a better job. in addition, one needs 

to be concerned that if a job change occurred and then if 
a subsequent termination occurred, it could be due to the 
client's match to the new job within the company, and not 
to the job for which the original match was made. Thus, 
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one really needs to obtain ue CJMS measure of the new 
job-worker (natch. In this evaluation, this was not 
done and it is not definitely known whether the termina- 
tions recorded occurred with the client being terminated 
from the original job or a different job. The analysis of 
CJMS match and client indices versus retention could be 
in error, since it analyzes the retention versus the CJMS 
match Indices obtained, using the onginal job profile. 
This aspect of retention (or other post-placanent measure) 
analysis was never explicitly dealt with in either the 
New Jersey evaluation design or In the SER evaluation 
design. 

Sections 4 and 5 of Volume II of this report discuss, in more detail, 
the SER/Cleff experiment and evaluation. 
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This report was prepared for the Manpower Administration, 
U.S. Department of Labor, under Contract No. B2C 5415. 
Since contractors performing research under Government 
sponsorship are encouraged to express their own judgment 
freely, the report does not necessarily represent the 
Department of Labor's official opinion or policy. More- 
over, the contractor is solely responsible for the factual 
accuracy of all material developed in the report. 

(Note: This study was originally contracted for by the 
Office of Economic Opportunity. The contract was sub- 
sequently transferred to the U.S. Department of Labor. 
The contract number remained the same.) 
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1.0 INTRODUCTION 



The detailed analyses undertaken 1n this evaluation are presented 
in this, the second volume of the final report. The major topics 
covered are: 

1) The attitudes and opinions of the staff of the agencies 
In New Jersey that participated In the original OEO-Cleff 
experiment. The organization of these agencies and the 
manner in which they operated the Cleff Job Hatching 
System Is also discussed. (Section 2) 

2) Analyses of the New Jersey agency's activity data obtained 
as part of the OEO-Cleff experiment. This will also in- 
clude a review of the ADP-PDS documentation and analysis 
of the New Jersey-Cleff experiment. (Section 3) 

3) The organization and operation of the SER-Cleff experiment, 
(Section 4) 

4) Analyses of the data obtained from the SER-Cleff experiment. 
The primary analyses to be presented Involve the determina- 
tion of the effectiveness of the CJMS In predicting the 
resultant retention of job-client matches. (Section 5) 
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THE HEW JERSEY/CLEFF MATCHING SYSTEM EXPERIMENT: 
AGENCY OPERATIONS AND STAFF ATTITUUES AND OPINIONS 



As has been discussed earlier in this report, the evaluation of 
the New Jersey/Cleff experiment was the original purpose of this 
contract. Since we have already discussed the events that led 
to the changes in the scope of this contract, we will not repeat 
the story here. However, before we present the analyses that 
were done relative to the evaluation of the New Jersey experiment* 
it is important that the reader keep the following points in mind: 

1) No quantitative data regarding the effectiveness of the 
Cleff Job Matching System was obtained from the New Jersey 
experiment, 

2) There were substantial differences in the extent to which 
the different agencies in New Jersey used the Cleff Job 
Hatching System and in the manner in which they used it, 

3) The OEO-funded experiment was run by the company that 

was marketing the Cleff Job Matching system. The company was 
Personnel Data Systems and was a subsidiary of a 
larger company, i.e,, ADP Incorporated. (The company 
that ran the experiment is hereinafter referred to as 
ADP-PDS,) The experiment was to provide data for use by 
Ultrasystems in evaluating the system. The experiment 
had been under way for approximately one year prior to 
the date the contract commenced. The capture of the re- 
quired data was the responsibility of AUP-POS, not of 
the agencies involved. The evaluation required other data 
that was to be obtained by the evaluation contractor* i.e., 
Ultrasystems, Inc. This data was essentially involved with 
obtaining measures of post-placement performance, i,e, 
retention, satisfaction, and on-job performance. The data 
was to be obtained via interviews with employers (e,g., 
mainly on-job supervisors) and clients. The capture of 
the infomation required so that the clients and employers 
to be interviewed could be identified and located was the 
responsibility of ADP-Pi)S, 

4) Whereas the New Jersey experiment did not provide any quan- 
titative data that could be utilized to evaluate the effec- 
tiveness of the Cleff system, it does provide some valuable 
insights into the realities of using this system within a 
manpower setting. This woS, in fact, one of the objectives 
of the original experiment. It was not the prime objective, 
however. 
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UUrasystems utilized this information in designing and 
operating the SER-Cleff experiment, 

5) It is UUrasystems ' opinion that the reasons for the 
failure of the New Jersey experiment should be studied 
by U.S. Department of Labor personnel who are involved 
in implanting evaluations of man-job matching systems. 

The ronainder of Section 2 is devoted to: 

1) a description of the operations of the agencies involved 
in the New Jersey/Cleff experiment, and 

2) the presentation of the attitudes and opinions of the 
staff of these agencies regarding the Cleff system. 

Section 3 presents the analysis of the data that was provided 
Ultrasystems regarding the New Jersey/Cleff experiment. 



2,1 OESCRIPTION OF OFFICE OPERATIONS AND STAFF REACTIOt^S: 

NEW JERSEY/CLEFF EXPERIMENT: BACKGROUND INFORMATION 

The Statement of Work set forth in the RFP for this evaluation 
included the following analytic questions: 

• Does the use of CJMS ppoduoe any dhcmges in the agency's 
operations? 

• Ai>e thepe "Hawthorne Effects" op othep interactions be- 
tween the CJMS and traditional agency operations? 

• ifliat do the placement agencies think of the CJMS? 

Thus the evaluation had, as part of its original design, the execu- 
tion of on-site interviews with staff members of the agencies par- 
ticipating in the Cleff experiments aimed at obtaining information 
so the above questions could be answered. One should bear in mind 
that the above analytic questions were formulated under the assump- 
tion that the participating agencies were using the Cleff system 
in accordance with the basic experimental design that had been 
developed. Shortly after Ultrasystems was awarded this contract, 
the participating agencies were visited by Ultrasystems staff so 
that firsthand knowledge could be obtained regarding the status of 
the experiment. The information obtained as a result of these 
visits was documented in a technical note submitted in August 1972. 
This document led to the eventual changes that were made in the 
evaluation's design. 
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The actual staff interivews were not done until approximately 
six months later, i.e., after the original evaluation design 
had been modified. 

The revised evaluation design promulgated by OEO proposed the 
following questions: 

1) Kow was CJMS implemented? 

2) To what extent was CJMS administered? 

3) What advantages of CJMS were noted? 

4) Uhat disadvantages of CJMS were noted? 

5) To what extent is the systen) useful for this type of 
an organization? 

Accordingly* Ultrasystenis set out to find what had happened with 
the CJMS in the participating agencies and what the staff of these 
agencies thought of the system. Specifically* our objectives were 
to; 

1) Obtain a description of how the CJMS was used by each 
agency participating In the experiment. This description 
includes* but is not limited to, the following k$y points: 

a) The criteria* If any, applied to the selection of 
clients by the agency and the reasons why this was 
done. 

b) The use made of the results of the CJMS scores. 
Basically* this involves determining if the system 
output was used as a counseling aid* as a matching 
criterion vis-a-vis job openings* and/or as a guide 
for job development activities. 

c) The effect agency con^'traints and operational require- 
ments had on system usage. 

d) The effect that local labor market conditions had on 
the system's usage. 
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e) The extent of CJMS usage by the agency's staff in 

terms of the number of counselors and/or job developers 
who have used the system and-^he percentage of their 
clients to whom SICLs were administered. 

2) The attitudes and opinions of the agency management and 
staff regatJing the usefulness of the CJMS. This involves 
the determination of staff experiences using tlie system, 
their assessment of the validity of the system's output and 
the relationship between the Cleff system's output and 
"test" results that th^ may use to assess client apti- 
tudes and job/training needs and potential. It also in- 
volves obtaining staff attitudes and opinions related 
to the system's overall advantages and disadvantages and 
obtaining their assessment of the systan's effectiveness 
in terms of eventual client job retention and satisfaction, 
employer and client feedback, and comparison with other 
matching and/or counseling systems. 

The following sections will respond to these objectives. 



2.2 INTRODUCTION TO THE NEW JERSEY MANPOWER AGENCIES 

INCLUDED IN THE STUDY 

In January 1973, Dltrasystems staff members visited the offices 
that were using the Cleff Job Matching System or who had partici- 
pated In the OEO/ADP Cleff Job Matching System experiment. These 
offices are listed In Table 2-1. It should be noted that over 
two years have elapsed since these Interviews were conducted. 
What has happened with these agencies and what they have done re- 
garding using the CJMS Is not known. 

Two of the offices shown In this table, Atlantic City Model Cities 
Vocational Rehabilitation (ACI1C) and Cape May Vocational Rehabili- 
tation (CMVR), are outreach offices for the Atlantic City Vocational 
Rehabilitation (ACVR) Agency and are supervised by this office. The 
Atlantic City Model Cities VR office was not visited. Staff nienibers 
of this office were Interviewed at ACVR where they were attending 
a general staff meeting. 

All of the offices except those in Atlantic City had "officially" 
stopped using CJMS at the time of these visits, since the experi- 
ment had reached the end of the OEO funding period. Several offices, 
however, still had CJMS terminal equipment on site and were able to 
access the computer so they could still perform CJMS scoring and 
matching. The Atlantic City Vocational Rehabilitation agencies 
had elected to continue using CJMS and had budgeted funds for CJMS. 
This was the only agency that had elected, at the time of the inter- 
views, to purchase the system and allocate funds for its operation. 
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TABLE 2-1 

CJMS PARTICIPATING AGENCIES AND AGENCIES INTERVIEWED 



Agency 
Number 


Abbrev. 
Title . 


Agency Name and Address 


1 


BICC 


Business and Industrial Coordinating Council 
50 Branford Place 
Newar^Ct New Jersey 


2 


ACMC 


Atlantic City Model Cities 
27 N. Massachusetts Ave. 
Atlantic City, New Jersey 


3 


ACVR 


Atlantic City Vocational Rehabilitation 
1601 Atlantic Ave. 
Atlantic City, New Jersey 


3 


CMVR 


Cape May Vocational Rehabilitation 
Cape May Courthouse, New Jersey 


4 


JCVR 


Jers^ City Vocational Rehabilitation 
2853 Kennedy Blvd. 
Jersey City, New Jers^ 


5 


NBVR 


New Brunswick Vocational Rehabilitation 

63 Morris St, 

New Brunswick, New Jers^ 


6 


PAMC 


Perth Amboy Model Cities 

(Diagnostic and Employment Center 

State of New Jersey Rehabilitation Comm.) 

262 State St. 

Perth Amboy, New Jers^ 


7 


TOPS 


Typing & Office Preparatory Skills ' 
Training Program 
506 Park Ave. 
Hoboken, New Jersey 


8 


FOCUS 


Field Orientation Center for 

Underprivileged Spanish 
469 Broad St. 
Newark, New Jersey 
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At least two other -agencies indicated satisfaction with the 
system and were certain they would purchase it; however, they 
were in the midst of funding crises which constrained them from 
purchasing the system. Whether they did or not is not known. 
Two other agencies had stopped using the system and had no in- 
tention of expending their cwn funds to purchase the system* 
and the remaining agencies had not arrived at a decision as to 
whether they should continue or terminate the system. 

Response to the system in all agencies visited was strong and 
it appears that, conceptually, CJHS has much to recommend it. 
However* mistakes were made in its introduction and operation 
which placed constraints upon its acceptance. Where these con- 
straints did not work against acceptance* imaginative use was 
made of CJMS. In some agencies, however* these constraints 
acted to obviate acceptance which in turn caused the user staff 
to dwell on these constraints to the detriment of the system. 



2.3 GENERAL DESCRIPTION OF THE AGENCIES INVOLVED IN 
THE NEW JERSEY CJMS EXPERIMENT 

The specific components of the Cleff Job Matching System were 
discussed in Vol. I and will therefore not be repeated here. 
However, it may be helpful to the reader In understanding what 
follows to get a general idea of the procedures to be detailed. 
Generally* each agency involved in the CJMS experiment operated 
in the same manner as they did before the experiment began* I.e.* 
before they began to use the CJMS. No attempt was made by any 
of these agencies to change the typas of clients they would seek 
to serve as a result of the use of the CJMS. Each agency tnade 
its own decisions regarding how it was to be decided that a given 
client took the Self Interview Checklist (SICL). All agencies 
were aware of the possibility of administering the SICL orally 
but> to our knowledge* only one agency ever continued to do this 
after an initial experience with this approach. 

The SICL's were usually administered within the agency by agency 
personnel* and the SICL responses were then transmitted by tele- 
type or mail to ADP-PDS for processing. The resultant scores 
were transmitted back to the agency* usually by the same means 
as was used to input the original responses. 

The Job Outline Checklist (JOCL) was utilized by only one of 
the agencies. Thus* for the other agencies* the SICL's were 
matched against the Job Cluster Register rather than against 
unique outlines obtained by agency personnel. 
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Each agency was given initial training in the use of the CJHS 
and could then continue to contact ADP-PDS to obtain answers 
to additional questions. In addition, several follow-up trainr 
ing sessions were provided. Each agency had considerable freedom 
in determining how and to whom to administer the SICL. Thus, the 
manner and order of administration of the SICL varied between 
agencies and between different staff members at a given agency. 

These very general patterns apply to both the Vocational Rehabili- 
tation and Community Agencies involved in the experiment. The 
following will first describe in general terms the operations of, 
each of the agencies Independent of their operation of the CJHS. 
This will then be followed by a description of the manner in 
which they administered the Cleff Self-Interview Checklists. 



2.3.1 The Community Agencies 

Each of the' three community agencies has its own separate and 
individual mission, although all are concerned with assisting 
the disadvantaged within their service areas to become vocation 
ally viable and to obtain jobs. Each of the agencies visited 
was faced with cutbacks in operating funds and, in turn, cut 
down on the number and variety of services it is able to offer. 



2.3.1.1 Business and Industrial Coordinating Council (BICC) 

The Business And Industrial Coordinating Council (BICC) is spon- 
sored by a coalition of businesses in the Greater Newark area. 
This was one of the first agencies to begin using CJHS and It 
participated in early development. BICC's experience with the 
evolving CJHS was prcHnisIng and encouraged OEO to fund the CIHS 
experiment. In general, BICC serves a primarily black disadvan- 
taged cormiunlty In Greater Newark. It acts as a liaison between 
business and the community for the purposes of increasing the 
availability of training programs and employment opportunities 
for members of the minority cormiunlty. It also assists local 
businesses and public agencies in efforts to Increase minority 
representation on their staff in compliance with EEOC criteria. 
In addition, BICC tries to rally cwimunity support for obtaining 
improventents In the Newark School System, day care availability, 
and number and quality of training programs. In direct support 
of these ends, it provides employment counseling, referral, job 
development, and some training services to the cormiunlty at large. 

CJHS was Introduced Into the agency in 1970, and, at that time, 
employment services were considered to be a minor function of 



The agency descriptions have been written In the present tense. 
However, we are reporting what the procedures were at the time of 
our visits. What they are now we do not know. 
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the agency. Concurrent with the use and development of CJMS. 
anployment services began to assume a larger role In the agency's 
operation until, as of the time of Ultrasystenis' visit to the 
agency, it became a major function of the agency. This change In 
enphasis cannot be entirely attributed to CJMS. since the agency 
is presently experiencing difficulty in obtaining operating funds 
from its sponsors as well as other sources. It had gone from a 
paid staff of fourteen to two persons at the time of the visit. 
OEO's funding of the CJHS experiment in this agency has undoubtedly 
influenced the present emphasis. 

The agency's director and counselor were Interviewed, as well as 
a member of its executive board and several former counselors who 
are no longer with the agency. It is interesting to note that 
the former counselors, although no longer associated with the 
agency (due to budget cutbacks), had very favorable opinions about 
CJMS. 



2,3.1.2 Field Orientation Center for Underprivileged 
Spanish (FOCUS) 



FOCUS Is comparable to BICC, but Its emphasis is primarily directed 
to Newark's Spanlsh-speaking population. It has been In existence 
for approximately five years and participated in the OEO/CJMS experi- 
ment from April 1972 to November 1972, FOCUS provides non-stipend 
training, employment services, general community services, and repre- 
sents the Spanish-speaking population of the Greater Newark area in 
matters of housing, public health, public assistance, and civil 
rights. 

Approximately 90 percent of FOCUS'S clients have less than a ninth- 
grade education, and a corresponding percentage do not speak English 
or have limited fluency In English. Of this percentage, approximately 
15 percent are functionally Illiterate or cannot read or write English 
or Spanish, Although 50 percent can read some English, the training 
provided Includes typing, clerical and secretarial skills, English 
As a Second Language (ESL), and a general educational development 
course leading to a high school equivalency certificate (GED), The 
training Is offered In both day and evening sections, FOCUS is 
unable to pay a stipend to students In these programs and reports 
difficulty in getting clients to enroll In them. Clients requiring 
training but who are unwilling to enroll In training without stipends 
are Ceferred to TEAM or the Dnployment Service, FOCUS provides 
interviewing, counseling, job development, and referral services 
for their clients. In addition, FXUS Is able to develop a few 
OJT jobs, but their experience In this area Is very limited. 
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2.3.1.3 Typing and Office Preparatory Skills Training Program 
(TOP?) - 



TOPS is a training center located in an old school building in 
Hoboken, New Jersey. It provides MOTA training in office and sec- 
retarial skills, as its name implies* and some counseling and job 
development services for its students. Since it is primarily a 
training institution* services are restricted to those accepted 
for training and are primarily training related. Clientele is mainly 
Spanish speaking and classes are offered in both day and evening 
sessions. 

TOPS accepts as students individuals referred by CEP* Model Cities* 
and other public programs and walk-in applicants who meet appropriate 
criteria for the disadvantaged. 

TOPS has used the CJMS to supplement its regular screening process; 
viz.* applicants for training are interviewed* tested to determine 
educational level (Metropolitan Achievement Test) and interest 
(CJMS)* and accepted or not accepted for training. Those individuals 
not accepted for training are referred to other agencies for service 
If appropriate* and those accepted for training are assigned a class 
and training period. Persons referred by agencies* such as CEP* 
are accepted and assigned training regardless of findings during 
screening as a result of contractual obligations with these agencies. 

The agency at the time of our visit was not using CJMS* since the 
OEO experimental funding had ceased. 



2.3.2 The Vocational Rehabilitation Agencies 

Six of the agencies visited are actually State Vocational Rehabili- 
tation offices of the New Jersey Rehabilitation Commission. With 
the possible exception of the Atlantic City Model Cities (ACMC) and 
the Perth Amboy Model Cities (PAMC) offices* office procedures in 
each of the offices follow the same general pattern. That is* intake 
consists of walk-In traffic* referrals from private agencies (church 
groups* Salvation Anny* etc.)* and frmn public agencies (the local 
welfare department* the Employment Service* etc.). At intake* appli- 
cants complete a Vocational Rehabilitation application. The form 
is then reviewed for completeness. The applicant is then referred 
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to a counselor who will review the application and ai)y related 
forms {such as the referral form from the Welfare Department), 
The counselor will then conduct a screening interview to determine 
if the applicant qualifies for service and to make an initial 
determination of the type of service the applicant requires. 

To qualify for service, an applicant must generally fall within 
specified poverty guidelines and have some physical or mental prob- 
lem which prevents the applicant from competing successfully in 
the labor market or interferes with his adjustment to society {e.g., 
drug addiction). In addition, the applicant cannot be receiving 
or be eligible for service from another agency {such as the Veterans 
Administration) and must have a problem capable of being success- 
fully resolved. 

In the case of the Vocational Rehabilitation agencies associated 
with Model Cities programs, the applicant must also reside within 
the Model Cities area. Applicants who do not qualify for service 
will be referred to whatever community service the Vocational Rehabili- 
tation counselor thinks will help them. Applicants who seem qualified 
can then be referred to a diagnostician for indicated physical or 
mental screening. If, based on the diagnostician's report, the appli- 
cant is still qualified for service, the applicant will be accepted 
as a client of the agency. Service to the client can only be termi- 
nated if the client successfully completes the plan of service or if 
the client's circumstances change sufficiently and the client no 
longer meets the agency's eligibility requirements {e.g., moves to 
a different area). 

Diagnostic services include such services as a dental examination 
by a dentist, a physical examination by a qualified doctor, a 
psychiatric examination by a psychiatrist or a psychological examina- 
tion by a psychologist or a psychiatrist. In addition, the appli- 
cant can be referred to other specialists such as an ophthalmologist 
or audiologist for indicated diagnostic services. 

All the offices, with the exception of the Perth Amboy Model Cities 
(PAMC) office, contract with outside professionals for diagnostic 
services. PAMC alone has Its own extensive vocational evaluation 
unit and, as such, was considered to be a pilot project of New Jersey 
Vocational Rehabilitation. PAMC uses portions of the Singer/Graf lex 
Vocational Evaluation System and TOMER Evaluation System test series 
and also performs the standard ABLE and CALIFORNIA scholastic attain- 
ment tests. 

All other diagnostic testing, including psychological, is contracted 
out. Figure 2-1 is an example of the kind of diagnostic report pro- 
duced by Perth Amboy. 
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Figure 2-1. Diagnostic Report Form Used by Perth Amboy, N.J. 

Diagnostic and Employment Center (p. 1 of 4) 
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Figure 2-1 (continued— p. 2 of 4) 
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Figure 2-1 (continued — p, 3 of 4) 
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Diagnostic reports are sent to the counselor handling the case, who 
will, depending on reported results, take whatever remedial action 
is required. Such action could range from getting a client exten- 
sive orthodontic work to helping find a job. Job development at 
the Vocational Rehabilitation agencies is usually performed by the 
counselor handling the case and is usually performed on behalf of 
individual clients. Vocational Rehabilitation agencies utilize a 
caseload approach. 



2.4 CJMS - OPERATING EXPERIENCES 

The following sections will present descriptions of the agencies' 
operating experience with the Cleff Job Matching System in the manner 
of a case study. The reasons for this are manifold: first, there 
was so much variation in the agencies* experiences with CJMS and 
the kind of data they retained with respect to CJMS that It is not 
feasible to create a single standard against which to describe this 
experience. Secondly, some of the data offered to substantiate 
claims with respect to CJMS are inconclusive and raise more ques- 
tions about agency practices not related to CJMS than the data answers 
with respect to CJMS. And, finally, events occurred in and around 
the interviewing process which could not be resolved by the Inter- 
viewer and led the interviewer to question the information received. 

As a result of this latter point, the descriptions presented below 
will be supplemented with a highly subjective description and Inter- 
pretation of the events which caused the interviewers to question 
the information received. 



2.4.1 The Community Agencies 

2.4.1.1 Procedures for the Selection of Persons to Take 
Self-Inteicvlew Checklists fSICL's) 

All of the cofimunlty agencies accepted walk-In applicants and appli- 
cants referred to them by other agencies. Intake to the agency 
(TOPS) primarily concerned with training was weighted more toward 
applicants referred by other public agencies than was the case in 
the other two community agencies Involved in this study. Three 
different approaches were taken in deciding to acbninister the Self- 
Interview Checklist (SICL). 

In the agency serving a primarily Spanish-speaking population (FOCUS), 
applicants are given a handout which informs then that they can avail 
themselves of the Cleff Job Matching System if they so desire. A 
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copy of this handout appears as Figure 2-2. About 5 to 10 percent 
of the agency intake volunteered for the SICL. Individuals who 
speak no English or who have poor conmand of English were usually 
discouraged from taking the SICL since it was felt that regardless 
of the results individuals with language problems could not get 
most jobs. We were told that most people applying for assistance 
at the agency were occupationally set and that this worked against 
their volunteering for the SICL. It was also felt to be bad form 
to explain how long it took to complete a SICL, since the time 
factor alone would discourage people from volunteering. 

At BICC> the CJMS was made an integral part of their service. Appli- 
cants entered, completed applications, and were assigned a counselor. 
The counselors then engaged the client in interviews to determine 
the client's service needs and to identify the factors which might 
support or interfere witKthe client's attainment of an objective. 
The client would then be offered the SICL after several such sessions* 
or as many sessions as the counselor judged necessary to obtain a 
basic understanding of the client and his needs. The counselors 
felt that this extended period of counseling was necessary in order 
to understand the client well enough so that they could allay any 
possible client fears with respect to entering a "testing" situation, 
and So that they could adequately answer any questions the client 
might have during the administration of the SICL. 

The remaining coimiunity agency* TOPS* which was primarily involved 
in training* gave a Metropolitan Achievement Test and an SICL to 
virtually everyone who completed an application. The results of 
the SICL were then used by the agency as part of their screening 
procedures to assess who would benefit from the training provided 
by that institution and to determine course assignments. The agency 
was obligated to accept individuals referred by such organizations 
as the CEP and place them in programs specified by the referring 
agency. Such individuals were also required to take an SICL and, 
although the results could not be used for normal screening* they used 
the results to detemine the individual's motivation for training 
and to validate; the course assignment. If such an individual, accord- 
ing to the SICL results* was obviously unsuitable for the agency* 
the agency would contact the referring agency and attempt to have 
the individual reassigned to different training or have the referral 
withdrawn. 

When asked by Ultrasystems* none of the staff members present remem- 
bered a referral withdrawn as a result of the initial determination. 
^k)wever, the staff members did say that these individuals usually 
dropped out of the training of their awn accord. 
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GIVEN TO CLIENT, WHO DECIDES ON TAKING SICL OR NOT: 



FIELD 0ItIBNarATX'5>: CBKTBR TOR THE VNDCRParYJH F'^EO SPANISH 

'P.O.C.U.S. NSWARK^ INC. 
469 BROAD STREET, NEWARK^ NEW JERSEY 07102 
TEL. 624-2528 - 29 

AT^TONIO PEREZ WILLIAM MATOS , JR. 

'^cutive Director Chairman 



P.O.C.U.S. inicia un Programa nuevo creado para resolver 
el problema de adaptacion al trabajo. Nuchas veces la& persor 
^ no duran mucho tiempo en un trabajo, por la sencilla razon de que 



el trabajo que tienen a cargo no es de su interns o porque no 

;i^lifican para continuar despues del tiempo de prueba* 
T^nemos un sistema moderno llamado "The Cleff Job Matching 
System** nuevo invento creado para resolver este problema de 
adaptacion porque le ayuda a descubrir sus cualidades para 
conforme a ellas buscar un trabajo mas adecuado para usted. Si 
desea aprovecharse 4e este sistema moderno, puede pedir in-* 
''^ormacio^n al Sr. Castillo, en esta misma oficina de P.O.C.U.S* 
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Figure 2-2. FOCUS Introductory Letter Informing Applicants of the 
Avallabnuy of the Cleff Job Hatching System 



2.4.1.2 SICl Administration 



The Self-Interview Checklist imposes very few restrictions on the ad- 
ministrator or respondents and appears to be a highly flexible 
instrument. The two booklets which comprise the preference and experi 
ence interviews of the SICL may be administered as an oral inter- 
view or as a written interview in English or Spanish. Ony a por- 
tion of the entire SICL need be administered and the administration 
can be closely or loosely supervised. Whether or not any of these 
features are used appears to be dependent upon a combination of fac- 
tors involving the agency's mission, the administrator's personality 
or cralning^ the level of technical assistance provided, the use 
to which the results will be put to, and a number of factors involv- 
ing the respondent. ^ 

If the SICL administrator suspects the respondent may be function- 
ally illiterate in English, the respondent is asked to read and 
explain a reading paragraph which contains many of the terms used 
in the SICL. Figure 2-3 shows a copy of the administrator's instruc- 
tions and the reading paragraph. If the respondent is unable to 
complete this task to the administrator's satisfaction, oral adminis- 
tration is indicated. TOPS will not administer the SICL orally be- 
cause th^ feel that a person who is functionally illiterate cannot 
be successful in their vocational programs. They are consistent in 
this approach in that th^ administer the Metropolitan Achievement 
Test separately to determine the respondent's educational level. 
TOPS alone of all the agencies has prepared a Spanish language ver- 
sion of the reading paragraph. 

Of the three community agencies, only BICC has administered the SICL 
orally, and this has been on a very limited scale. Oral adninistra- 
tion of the SICL requires up to a full day of the respondent's and 
the administrator's time. The CJMS manual calls this method of ad- 
ministration a "Direct Rating Format." Other than stating that a 
rating is required for "every activity (16) in every group (15)" 
for each interview administered, no instructions are supplied on how 
to administer the SICL orally. Counselors at BICC have tried, on 
their own, several methods of oral administration but found none of 
the methods very satisfactory. 

The Spanish-speaking clients of TOPS and FOCUS are representative of 
Cufcan, Puerto Rkan, and Latin American cultures. There are dialec- 
tical differences in each culture and some words do not have sinnlar 
meanings in all cultures or in formal or informal usage in the same 
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PERSONNEL DATA ^;YST£MS, INC. 

d43 MAOlSOll AVL'IfUC 
NtWVDftK.N.V. tOOlT * Ulll €H.t4TT 

CLEFF MATCH IMG .QYflTEM 



Instructions to Counfjftlor - Intex'vicwer 



huk the candiclatc to read the READIHG PAKAGnAFH. 

Say* "Wg in this office arc writing a brochure for our 
cuntoineroi the 30b applicants liUc you, and wo want to 
mnl:c norc that v;o are u^ing words and lancuage which 
maltcn sense to you* V7e would like you* to rocid it aloud^ 
then tell us if you tl)ink wo can Make it hcttoti and 
ba-; to do that." Or words to that effect. 

Permit the applicant to read it aloud without inter** 
fcrcncoi but holp if ho ha^ trouble, in a sympathetj^ 
way of course, to get him through. 

Keo)> note of tho nurnber of words he cannot accurately 
rciK^ . i-!isprono\incing a word docn not necG&£;arily .mean 
ho cnni;ot Jrcad it* Ucc your o:/n judgment. 

Sf, V^'V*^^. ? ., !'- ' cannot: r-:?ad fivp or vr.QTo v/n ^dj: in l:ho 
pnrc -^r r.rji , th r:; uo not r^^;"or hin for tho »V-i:^ d^:ra 0,T,C.Ti. 
Jm^tcaui you m,iy want to consider aciminirs'i-cring the Direct 
Rating Format of the S.I.C.L. to him ox'ally* 

1€ you ndminisLur eitliur tnu Sbun<}ai*<} or Liie Diircct nctln^g 
Forinut of the S.I.C.L. < introduco the applicant to tho 
sy£:tom - tell him about increasing the potuntial £or a 
good job match. 



Figure 2-3. CJHS Rtadlng Paragraph- 
Interview Instructions 
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'i^jcrc arc rriany hinds of people, and many Kinda of jobo* 
So:ae poopl<! operate raid graders, paint: outside of 
liourjors, ooll groccricri in a st:orc, or rai^c-ir autor.obilas. 
Othor people may proctch cerraonn in church, take nciic;]ibor- 
hooO survoyo/ demonstrate automobile p'artii for :>alc, or 
do reifoarch on hifitoriual evonts. And hho jobii asik 
^pooplcj to do varioufi things; for example, Ceod children, 
]ay brickii, v;ork vuth a micro,';copc^ interview job Ctin- 
didatos, and maybe oven do mechanical drr.ving- 'ilicre 
i/j no te)iing v,*hat a job might demand of a per;:on. Wo 
l:now of a job where the worhor v;as uskcid to cnporviso 
cloaning a building, read complicatod instructions, road 
gamjcs on machinery, measure food ingredients, and 
catalogue mail order morchandisC!. He fonnd the job 
interesting because hfe was asked to do so many different 
things. 

Figure 2-3 (continued) 
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culture. An additional problem arises in that some American English 
SICL phrases have no meaning in Latin culture and are not strictly 
translatable into Spanish. These and other problems have resulted 
In several attempts to develop a Spanish language SICL that is uni- 
versally usable. 

The TOPS counselor stated that the current version was a good trans- 
lation and that they used the Spanish language SICL. FOCUS preferred 
not to administer the Spanish language SICL, since they felt it was 
futile because there were so few jobs that would accept individuals 
who spoke only Spanish. It was later determined that one of the 
individuals hired to translate the SICL into Spanish was the TOPS 
counselor. 

At TOPS, the applicant would be scheduled for the SICL immediately, 
and the written SICL was usually given on a mass basis to about 
twenty persons at a time. TOPS felt that it required, at a minimum, 
one administrator or assistant for every ten persons taking the SICL. 
During one period, they had the use of volunteers from a local college 
to help monitor the administration of the SICL's. The Metropolitan 
Achievement Test was administered either before or after the SICL, 
and staff members said they noticed no differences as a result of 
this proximity. 

The administrative procedure was for the TOPS counselor to explain 
the method of completing the SICL to the respondents and to satisfy 
herself that they understood the procedure. The purpose of further 
monitoring by TOPS staff was to keep order in the roan and to be 
sure the respondents were completing the SICL properly. If a re- 
spondent did not understand a phrase on the SICL, It might be ex- 
plained to him, but usually it was not. TOPS felt that if the 
respondent did not know, for example, what a bulldozer was, then the 
respondent did not have any preference (or experience) in that area. 

As each booklet was completed, a monitor would check it for canplete- 
ness and understanding. The booklets would then be mailed to ADP-PDS 
in New York for scoring. The scores and cluster matches would be 
received in approximately three to five days from the date the SICL 
was completed. 

On the other hand, a less mechanistic approach to SICL administration 
was taken by BICC. Groups of up to ten individuals would be scheduled 
for the SICL. Counselors took turns in giving the Sia's each day 
so they administered the SICL's to each other's clients. The pro- 
cedure for taking the SICL would be explained and the respondents 
requested to complete the first page of the SICL. While this page 
was being completed, the counselor would spot check individual 
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responses by questioning the respondent about his choices. This 
type of spot checking would be continued throughout the Interview 
period. Some counselors stated that respondents had difficulty 
in understanding the four categories of selection (Most* Least* 
More, Less) used by the SICL. In these cases, they might provide 
their own interpretations of these categories. For example* Least 
might be explained as "Pick the phrase you have never done and 
don't think you will ever do." 

If, after the procedures were understood, the respondent had ques- 
tions about the meaning of ai>y of the phrases* he could ask the 
counselor to explain. The counselor* who could determine which 
Dimension of Work each phrase referred to, would explain or Inter- 
pret the phrase to the respondent. To BICC counselors* It was 
considered necessary that the respondent understand as many phrases 
as possible In order to complete the SICL properly. After the SICL 
was completed* the raw SICL results could be entered Into an on'Slte 
computer terminal for processing by time-sharing computer. Turn- 
around time was minimal and the respondent's SICL results and JOCL 
and/or Cluster Matches could be received quickly. 



2.4.1.3 Results — Interpretation and Use 

The results of the SICL were received In a variety of ways. At FOCUS, 
the SICL scores were sent to ADP-POS* who entered the scores into 
the computer, interpreted the results, and sent the computer results 
and interpretations to FOCUS. FOCUS reviewed the interpretations and 
then prepared index cards summarizing the interpretation. Figure 2-4 
displays the Interpretation font) prepared for FOCUS and the index 
card prepared from that form. Figure 2-5 is a copy of the computer 
printout from which the Interpretation was derived. The name of the 
respondent has been obscured 1n both figures to protect his privacy* 
but both figures refer to the same respondent. 

The respondent's scores in Figure 2-5 begin at the top with his 
Occupational Adjustment Factor (now called Occupational Adjustment 
Index) The respondent's scores end with the line of numbers beginning: 
"COM; 3-2 3 .,.." These scores are maUhed to clusters. Each cluster 
the respondent has been matched to begins with the titlev "Match #xx" 
and ends with the first line of numbers under the line beginning "TT T2 
Note that a cluster Is not a job but a type of benchmark score which 
r^resents a related set of jobs (related by CJKS). 

The numerical scores for the respondent are translated by ADP-POS 
to ranges {high, low, and neutral) on the Interpretation sheet for 
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each of the CJMS Dimensions of Work. Conments related to this 
translation are found at the bottom of the Interpretation form and 
a few of the kinds of jobs the respondent might do well are found 
below that. Note these are not available jobs but suggestions as 
to the kind of job to develop for the respondent. It appears that 
the Interpretation* especially If only the Index card summary Is 
considered) needs a lot of Interpretation Itself. This limitation 
appears to be a strength of the system In that it flags possible 
problems with respect to a person's adaptation to work* gives clues 
as to why the respondent failed to adapt, but places the burden of 
finding out why squarely on the shoulders of the counselor. 

TOPS receives computer printouts similar to that shown In Figure 
2-5. TOPS does Its own Interpretations and works directly from the 
computer printouts. 

Figure 2-6 shows a copy of the computer printout for BICC. It Is 
similar to the printout shown for FOCUS but differs In one Important 
respect: the matches are to actual Jobs and not job clusters. BICC 
alone of all the agencies using CJHS (Including the Vocational Rehabili- 
tation agencies) has developed jobs which have been CJMS-descrlbed and 
has entered these descriptions to the CJMS job bank file. 

BICC has a teletype computer terminal. When the SICL's were completed, 
the teletype operator would punch the scores directly from the SICL 
booklets and transmit these scores to the on-line CJMS computer. 
Barring terminal or computer breakdowns* the computer calculates 
the adjustment and match indices* the dimensions of work scores* and 
matches these scores to the comparable job-related scores. This data 
Is returned in match order within minutes after It has been entered. 

BICC does not purge Its CJMS Inventory of jobs when the order Is 
filled. Instead, the CJMS job profiles are retained on file and used 
as a job development source and as a ccuinsellng tool. The BICC jobs 
listed with CJMS are not normally available to other agencies in the 
system; however, BICC has shared Its Job openings with FOCUS. 

FOCUS does not appear to have made much use of the SICL results. 
Th^ called In the respondent and went over the results with him. 
They found most people who took an SICL anxious to find out the 
results but were unable to make much use of the results. Lack of 
facility In English and no training stipends limited clients In the 
training and jobs they would accept or could expect to get. FOCUS 
felt that the SICL was useful In helping some clients make up their 
minds for the future but, due to the limited resources available to 
them (or FOCUS), the SICL was not applicable. Most referrals of 
clients to jobs were made on the basis of the client's prior experi- 
ence, not the SICL. 
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TOPS was also limited in how they could use the SICL. The SICL re- 
sults were used along with educational achievement tests to screen 
persons for the training programs TOPS offered. People v^o quali- 
fied for their service and who indicated clerical or secretarial 
inclinations in the SICL were accepted for their programs. Others 
were referred to another agency that mighi be able to help them. 
The only exception to this were individuals referred by the CEP. 
TOPS is obligated to accept all CEP referrals and must generally 
place these individuals in specified training. TOPS felt they could 
tell from the SICL whether the person referred by the CEP was moti- 
vated sufficiently to cwnplete the training {in addition to being 
suitable for such training). Although TOPS tried to have the CEP 
withdraw or change what the SICL Indicated was an Inappropriate 
referral, they were unsuccessful. TOPS felt this CEP refusal to 
change or withdraw the referral was somewhat burdensome* since the 
Individual generally dropped out before con^letlon of training. 

Once an individual was accepted for training, TOPS provided counsel- 
ing services designed to help the client continue in the training 
and to help the client prepare for a job In the area in which he 
was being trained. The counselor felt that the SICL results, espe- 
cially the "People-oriented" parameters, were very useful In counsel- 
ing at this point. Another benefit cited for CJhB was that It forced 
TOPS to improve their training. Since CJMS helped TOPS pick the 
best people, the training standards had to be raised a corresponding 
amount. 

TOPS staff stated that they felt very definitely handicapped without 
CJMS and, If they had the money, would start using it again. 

BICC made the most extensive use of CJMS. They used CJMS as a 
basis for selecting people for training and jotss. The counselors 
used SICL results In counseling sessions and felt that the infomsa- 
tlon provided by the SICL considerably shortened the time required 
for counseling. CJMS was used as a diagnostic tool and was thought 
to be more useful than the Kuder or Minneapolis. It helped them 
reduce categorization errors such as occur when a client neglects 
to tell the interviewer of some significant past experience or 
when a single DOT code Is assigned to an Individual with a broad 
vrark experience. 

' In one Instance, BICC administered the SICL at nearby Orange High 
School. The students were not Interviewed beforehand and the SICL 
was administered en masse. BICC stated that the school found the 
SICL to be accurate and useful for providing the students vocational 
counseling. 
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2.4.1.4 Caveats -- Interview s 



None of the agencies maintained for themselves statistical data 
specifically designed to test or evaluate their experience with 
CJMS. What statistics and information was kept was introduced 
by ADP-PDS as part of the OEO experiment. Statements made about 
CJMS by the agencies probably use personal experience as a base of 
reference and this personal experience seems to weight rather 
heavily in the agency's evaluation of CJMS. 

BICC, for example, was the first of the agencies to use CJMS. They 
were instrumental in introducing CJMS to OEO and participated In 
at least two revisions to the system. BICC also appeared to have 
received the most attention from ADP-PUS in the form of technical 
assistance. This history could have given BICC almost a proprietary 
interest in seeing CJMS accepted. 

BICC has administered more SICL's than all of the other seven agen> 
cies combined* accounting for almost 60 percent of all SICL's In 
the experimental group. In addition, BICC is the only agency to 
JOCL jobs and to establish a CJMS job bank. It has placed more 
people from the experimental group into jobs than any of the other 
agencies {approximately 42 percent of the total; see Section 3). 

FOCUS feels that it has a special problem with Spanish-speaking 
applicants who do not have a fluency In English. Their attitude 
seems to be that unless their clients attain fluency In English, 
they win be restricted to a small set of jobs and that very little 
can be done until the language barrier Is surmounted. 

TOPS maintains success statistics, but this data uses as a base 
individuals accepted Into their programs. Furthennore, at the time 
of this interview, there wa« only one counselor. This 
counselor had been hired by ADP-PDS to translate the SICL into 
Spanish. This latter fact may have Influenced her attitude toward 
the SICL. 



2.4.2 The Vocational Rehabilitation Agencies 

2.4.2.1 Procedures for the Selection of Persons to Take 
Self-Interview Checklists {SICL's^ 

Intake for the Vocational Rehabilitation agencies Is primarily a 
result of referrals from other programs such as the local welfare 
departments, probation department, and Model Cities agencies. 



11^29 



ERIC 



145 



Applicants complete an application and then see a Vocational 
Rehabilitation counselor who will assess the Indlvlduars ellgl- 
blllty for service and determine what services to offer the client. 
The counselor may require the client to undergo any of a number of 
diagnostic services In order to ascertain appropriate remedial 
services. CJMS was accepted by the agencies in this study as an- 
other diagnostic service to add to their repertoire. 

The counselor thus could select the CJMS SICL as one of the diag- 
nostic services to offer a client. The SICL could be administered 
at any time during the client's tenure of service at the counselor's 
discretion. The counselors are therefore the to the use of 
CJKS In the Vocational Rehabilitation offices. Iti Ultrasystems' 
opinion* ADP-PDS Initially had problems communicating to the Voca- 
tional Rehabilitation offices selected to participate In the experi- 
ment what the experiment was about and v^om It was to serve. 
About two months after the Installation of COMS* the Vocational Re- 
habilitation offices held a meeting to clarify these Issues* and 
the New Jersey State Vocational Rehabilitation Director Instructed 
the office managers to use the system. No additional staff was 
allocated to administer or process the SICL's. This caused objec- 
tions, since the offices already felt that they were understaffed. 
ADP-PDS conducted at least three training sessions for the staff of 
these offices, but failed to convince marty of the counselors of the 
value of the system as compared to other diagnostic services then 
available. These difficulties, plus related difficulties discussed 
below, posed a severe handicap to CJMS. The handicap was severe 
enough In one office (Jers^ City) that* except for some experimenta- 
tion* CJMS was not used. 

Another limitation to the use of CJMS In the Vocational Rehabilitation 
offices lies In how the offices perceive their mission. Th^ are 
not employment offices and have no full-time job developers. Job 
development Is usually performed by counselors through their own 
contacts or through referrals to other agencies such as the Employment 
Service. Individuals entering with medical problems tend to get 
services designed to alleviate the specific diagnosed complaint. 
Unless employment requires special considerations on the part of the 
employer, as with mentally retarded persons, or Is considered part 
of required treatment, as with drug addicts, or Is easily attainable* 
the client Is not usually given much help In finding employment. 
The client In this case Will be referred to the Employment Service 
or returned to the agency v^lch referred him in the first place. 
Thus, the Atlantic City Model Cities office felt that most of Its 
caseload "were medical problems and not employment problems." After 
the CJMS experiment ended, ACHC's service load, which excluded drug 
addicts and juveniles referred by Welfare, was expanded to Include 
these groups, and It was felt that the use of CJMS would Increase 
substantially In order to serve these people. 
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In addition, offices such as the New Brunswick Vocational Rehabili- 
tation (N8VR) had a large number of clients who were already work- 
ing but came in for medical services. Although their income was 
low enough to qualify them for Vocational Rehabilitation services, 
they were not normally provided employment services because they 
were already employed. Since they did not require placement help, 
they did not get an SICL. 

Further inhibiting the number of clients that would receive the 
benefit of CJMS were the problens of administration and the manner 
in which the results of the SICL were delivered. 



2.4.2.2 SICL Administration 

* 

The Perth Amboy Model Cities (PAMC) office was especially prepared 
to administer SICL's. This office is designated as a diagnostic 
and employment center and Is an experimental program of the New 
Jersey Rehabilitation Commission. It has testing facilities and 
testing technicians with which it can and does perfonn several 
commercially available vocational series. The counselors utilize 
information from the testing as input for their vocational counsel- 
ing. None of the other Vocational Rehabilitation offices did their 
own testing. As previously stated, the usual procedure for other 
Vocational Rehabilitation agencies was to utilize the services of 
outside commercial diagnosticians for whatever evaluation the coun- 
selors required. 

A similar pattern of SICL administration found in the community 
agenci'*" was found in the Vocational Rehabilitation agencies. That 
is, soiiit. agencies would administer the SICL mechanically providing 
the respondent with very little help in taking the SICL. Others 
were careful to Insure that the respondent understood every phrase 
of the SICL, There were no apparent difficulties In administration 
of the SICL as such. 

Problems arose simply because the SICL had to be administered. 
Someone, usually a counselor, had to take the time to administer 
the SICL. If the SICL was administered orally, it could take four 
or more hours. If the SICL was administered in the standard written 
manner, it required one to two hours of the counselor's time. The 
counselors stated that they did not like to take this much time 
from their caseloads to administer a "test." Furthermore, they 
were not used to administering diagnostic evaluations, since all 
such evaluations usually were performetJ by an outside source. This 
probl«n was mentioned by all the Vocational Rehabilitation offices 
except PAMC, which had internal staff specifically designated to 
administer tests for the agency. 
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The offices experlniented with various ways of alleviating this prob- 
lem but could not come up with satisfactory results, Atlantic City 
Vocational Rehabilitation (ACVR) managed to solve this problem when 
they purchased the CJHS system after the experiment ended. As part 
of the purchase agreement* ADP-PDS agreed to hire* tra1n» and place 
on site an Individual who could administer the SICL and take care 
of paperwork and Interpretation. The arrangement has been satis- 
factory for ACVR. The SICL technician administers the SICL for 
both ACVR and Its Model Cities satellite (ACMC). ACVR's second 
satellite office Is about forty miles distant in Cape May» New 
Jersey , 

The two counselors staffing the second satellite office administer 
the SICL themselves and send the booklets to ACVR for scoring and 
Interpretation. They have experienced no difficulty In admlnlsterr 
Ing the SICL> primarily because they have a relatively low caseload 
(most of their cases are presently employed) and because they only 
give 10-12 SICL's a month and are able to Integrate the SICL's Into 
their schedules. 

The Vocational Rehabilitation agencies have experienced some diffi- 
culty In getting the respondents to understand the phraseology of 
the SICL. As In the community agenclest the respondents are func- 
tionally literate; 1.e.» they can pass the reading test. CMVR 
solves this problem by simplifying phrases. For example, they will 
change the phrase, "Operate eyelet machine*" to "Operate a machine 
in a factory." Perth Amboy does this too, and It also explains the 
•ielection parameters, "Most, Least* More, Less" as "love, hate, like, 
dislike." 



2.4. R esults: Int erpretation and Use 

The Vocational Rehabilitation agencies had much more to say about 
the procedures and technicalities Involved with obtaining and using 
the results than did the community agencies. There are several pos- 
sible reasons for this. The major reason was the use of outside 
diagnostic services which could serve as a base against which CJMS 
could be compared. The following paragraphs divide the conments 
into three general areas: the procedures Involved in scoring the 
SICL, the interpreting of the SICL scores, and the use of the score 
results. 



2.4.2.3.1 Scoring the SICL 

Unlike most commercially available tests, the SICL is clerically 
complex and difficult to score. For all practical purposes, a 
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computer or similar device Is required to score the SICL. Some of 
the offices {PAMC» ACVR) had a teletype unit Installed, The tele- 
type unit could be connected by a telephone hookup to the General 
Electric Computer Time Sharing System on which CJMS was run. To 
score an SICL and obtain matches to clusters or JOCL's» the terminal 
operator would type the SICL responses directly from the SICL book- 
let. A paper tape would be prepared the same time. The operator 
would then dial the computer system on the telephone and Insert the 
telephone receiver Into the telephone coupler on the teletype machine 
when the computer's ready-time was received. The operator would then 
enter the requisite opening statements* run the previously prepared 
tape of the SICL» and receive computer calculated SICL scores and 
matches In return. 

Some offices such as N8VR and JCVR preferred to mall the SICL's 
to ADP-PDS and have them obtain the scores and matches from the 
computer, ADP-POS would then mall the results back to the originat- 
ing office. While the use of an on-site terminal Is an efficient 
method for scoring and mstching» It also provided the greatest source 
of complaints about the system. Computer breakdowns and terminal 
breakdowns, although apparently Infrequent* were highly visible to 
offices with terminals. Personnel had to be trained to operate the 
teletype machines* and, when the offices tried to switch typists, 
they Immediately ran Into problems due to the more complex nature 
of telecommunications as opposed to the relatively simple operation 
of a typewriter. Adding to these complexities were the problems 
associated with the equlj^ment In general. 

To demonstrate these problems* the teletype operator In Perth Amboy 
gave us a demonstration. She called the computer three times and 
hung up the first two, explaining that If the computer's "ready" 
signal was not loud (e.g., a poor telephone connection)* there would 
be problems later on. She then placed a previously prepared tape 
into the machine and transmitted the SICL scores as shown In Figure 2-7. 
Notice the left-hand margin Is not even. She then received the mes- 
sages shown In Figure 2-8, Notice the dark boxes at the left-hand 
margin. The first boxes should be a line of captions {Tl» T2t T3t 
.,, the second box should be a line of experience scores; the 
third box a line of preference scores; and the fourth box a line of 
combined scores. The boxes are actually letters and numbers being 
typed In the same space on the line. The misaligned left margin 
makes It difficult for the operator to detect typing errors. These 
are problems caused by a light teletype machine or by a machine re- 
quiring adjustn^nt. In the space of one hour, we were not able to 
get a message free of any overstrikes or clear enough to use, (It 
was a rainy day, which might account for some of the problems,) 
Two ways to get readable scores Involve requesting re-runs. One 
method Is to keep requesting re-runs until a clear message Is received 
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or an intelligible message can be constructed. The second method 
requires the operator to cause the scores to be received in two 
modes — tape and print. In the first method, computer and tele- 
phone charges are incurred for every re-run request. In the sec- 
ond method* if the fault Is in the printing, the tape can be re-run 
at a later time without incurring extra telephone or computer charges. 
Until we started running tapes to determine the cause of the print 
errors, the teletype operator did not realize that the second method 
could be used to get clear copy. 

Problems associated with the teletype were so severe that the PMV: 
operator complained that it used a significant amount of her time 
(she was also assigned other duties). NBVR* which initially had a 
teletype unit, started mailing SICL's to ADP-PDS for processing. 
ACVR probably would not have purchased CJMS unless a person was 
provided who could administer SICL's and operate the teletype. 



2.4.2.3.2 Interpreting the SICL 

A psychological or psychiatric diagnostic report usually consists 
of several pages of narrative describing tests given, tUe test 
results, results of personal interviews (including some background 
data which may or may not have been uncovered by tite counselor)* 
descriptions of clients' reactions during the examination period> 
summarizing conclusions, and suggestions for a future course of 
action. The SICL provides a set of experience, preference, and 
combination scores* measures of correlations and differences for 
these scores and, if requested* matches to jobs or clusters with 
appropriate scores and measures of correlation and difference. 
A manual is provided by ADP-PDS which defines each of the scores 
and the measures of correlation and difference and provides informa- 
on how to use these scores and measures. The clusters accompanying 
an SICL match suggest a number of jobs the client might do best. 

In the psychological or psychiatric narrative, much of the interpre- 
tation of tests and correlation of data has been done for th$ coun- 
selor by the examiner and this infonnation Is arranged so as to 
support the examiner's conclusions. Edch of the Vocational Rehabili- 
tation offices'* except for Perth Amboy, expressed dissatisfaction 
with the presentation of SICL results. 

The senior counselor at the Jersey City VR felt, that compared to 
the psychological evaluation, CJMS "was a joke. . . . [you] can 
get all the infomation that [CJMS] gets from the interview." Both 
the New Brunswick and Cape May VR offices complained of the time 
required to interpret the SICL. Each of the offices stated that 
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It required time to learn how to Interpret the SICL, experience 
In the job market In order to be able to relate the clients to 
the job market, knowledge of the Jobs actually available to their 
clients, and experience with CJMS In order to understand CJHS 
well enough to use It as a diagnostic tool. 

The Jersey City Vocational Rehabilitation (JCVR) office felt that 
CJMS was Inferior to the quality and sophistication obtainable 
from their consultants. NBVR similarly saw no advantage to using 
the CJMS because of the expenditure of their own time to administer 
the SICL, the quality of interpretation, and the time required to 
obtain results when they mailed the SICL for scoring. JCVR stated 
that they gave SICL's and psychologlcals to two clients and titat 
the psychologlcals were "much more revealing and Informative." 

Atlantic City VR had reduced the problem somewhat by having the 
ADP-PDS SICL technician chart the results (Figure 2-9 Is a sample 
of the form used to chart the results). The charting Is simply a 
clarification on the presentation of SICL results and counselors 
still expressed preferences for the psychological narratives. 

Perth Amboy MC presented the results of the tests performed by its 
evaluation section on the score sheet shown in Figure 2-1 which 
includes the evaluation of the result? of all the tests Including 
the SICL. The majority of counselors and evaluators felt insecure 
In interpreting the SICL's since they don't fully understand the 
specific meaning of the parameters or statistical base for determin- 
ing the validity of these parameters. One of the counselors stated 
that she had done an evaluation of the educational level of the 
words used in CJMS by a technique developed by Dr. Fry of Rutgers 
University and had determined that the SICL required a tenth grade 
reading level, not a sixth grade, as ADP-PDS claimed. 

Some counselors questioned how the SICL could be interpreted at 
all for some clients and gave as examples clients whose point of 
reference was so low that they had never heard of some phrases. 
Thus, some clients might actually prefer to "solder wire connec- 
tions" or "box cans on assembly lines," but, being Isolated In their 
community, have no idea what "soldering" is or what it would be like 
to work on an assembly line. The counselors agreed that lack of 
knowledge might indicate lack of interest, but they pointed out 
that each page of the SICL contained many such phrases. This, 
coupled with phrases such as "Administer Medicine" (as opposed to, 
say, "give medicine to sick people"), could result in too many 
passed phrases on an SICL, which In turn Invalidates the results. 

One office stated, in reference to CJMS, that they had never seen 
a test that was not racially or culturally biased. Another office 
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Figure 2-9. CJMS— Form Used to Chart SICL Scores 

11-38 1^:1 



questioned the validity of the SICL between the English and Spanish 
versions. They pointed out that translations were not always 
literal and that the difference between "make pizza" and "hacer pan 
y galleticas" (literally, "make bread and cookies") could result in 
significant differences in selections of Spanish-speaking clients- 
depending on which version they took. With respect to the Spanish 
language version* PAHC also pointed out the problems associated 
with dialect or jargon in translation. Thus, a Cuban might under- 
stand "cartuchos" to mean cardboard boxes* but a Chilean might 
think It meant cartridge cases. 



2.4.2.3.3 Using the Results of the SICL's 

Only the Atlantic City VR offices found any use for the SICL. In 
PAMC, one of the counselors stated that she used the SICL as it 
was intended to be used but, given the overall office response, 
her statement Is in question. The other three counselors in the 
PAMC interviewed stated that they made minimal use of CJMS* and 
one counselor Is quoted as saying, "I've learned to Ignore it for 
at least six months." PAMC evaluators and counselors feel that 
CJMS may be somewhat useful for Inexperienced persons with no idea 
of direction, but that they would rather spend the time interview- 
ing the client. 

New Brunswick VR counselors could not remember CJMS well enough to 
recall what use they made of the results and JCVR, with less than 
twenty SICL's administered* did not hava sufficient experience with 
CJMS to make active use of the SICL. 

The Atlantic City offices have been able to make extensive use of 
CJMS* although it appears that the counselors are still probing to 
determine the limits of the system. Cape May counselors* for 
example* can Interpret the CJMS results but feel they need more 
experience interpreting In order to be able to derive the maximum 
amount of information from the results. Nevertheless* they have 
found that CJMS results provide them with a direction in counseling. 
This helps them get Into effective counseling more rapidly, since 
it speeds up the fact-finding portion of the counseling period and 
helps avoid some counseling blind alleys. In addition* the Cape 
May counselors feel the CJMS results serve as a good tool for 
determining a need for a psychological or psychiatric examination. 
Note that the CJMS Is not used as a psychological diagnosis but 
rather as an indicator that such diagnosis may be warranted. 

Similarly, ACVR and ACMC counselors feel that CJMS gets them Into 
counseling faster by providing a shortcut to fact finding. These 
counselors feel CJMS has soirte advantage over psychological examina- 
tions in that It breaks down client/job-related characteristics 
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into more specific areas. This enables them to establish counsel- 
ing and vocational objectives faster. They find the CJMS especially 
useful for working with the mentally retarded, since they feel that 
standard I,Q, tests provide meaningless results, whereas the CJMS 
helps this client express his interests and vocational strengths. 

One counselor stated that she used the SICL as a quick I,Q, test 
to determine if a client should be sent to college, A psychologi- 
cal test is still conducted since this provides an I,Q, score which 
must be used to justify the decision for the record. In general, 
Atlantic City counselors liked the speed with trtiich tiiey could 
obtain usable results. They could get SICL results back in the same 
day or within a few days, whereas it could take up to a week before 
a client could meet with a psychologist and up to three weeks before 
the psychologist's report could be received. 



2,4,2,4 Caveat— Interviewing 

The objective of the interviewing was to determine how CJMS was used> 
what impact it had on staff/client relationships* and to gain insights 
into how the system could be used and whether these uses appeared 
to be effective. The purpose was to evaluate the system and not the 
agencies using the system or the services being supplied the system. 
These objectives held up in the interviews wit^ the community agen- 
cies but collapsed in the Vocational Rehabilitation agencies as it 
became evident that there were problems external to CJMS that were 
having deleterious effects on CJMS usage. 

It appears* for example* that CJMS was imposed upon t^e Vocational 
Rehabilitation agencies in the experiment and that they were virtu- 
ally ordered to use CJMS by their State Director, We have no direct 
evidence to support this conclusion other than cautiously worded 
statements from staff members which seem to imply this. In addition, 
it appears that some of the personnel responsible for introducing 
CJMS acted boorishly and tactlessly with Vocational Rehabilitation 
staff persons and that t^e manner in i^ich technical assistance was 
provided only made problems worse. 

In one office, for example, the office manager remembered CJMS with 
sufficient detail to be able to discuss the system. On a second \ 
interview visit to that office, the manager was absent and the inter* 
viewer attempted to find staff members who would discuss the system. 
Sane staff claimed not to remember the system and others were reluc- 
tant to talk about CJMS, giving brief, curt answers in response to 
interviewer questions. The interviewing in the office finally had 
to be terminated because of this resistance. One counselor did 
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remember CJMS, but her memories were mainly limited to an ADP-PUS 
training conference in which the instructor told "dirty jokes" which 
she took objection to. This conference and reaction was confirmed 
in another office. 

In another office, staff members canplained vehemently of the treat- 
ment they received from ADP-PUS staff members. They told of 1) brisk 
responses from ADP-PDS to their questions; 2) calling Ai)P-PDS offices 
in New York and not being able to get someone who could answer their 
questions or of not receiving answers after being told that someone 
from /U}p-POS would call back with an answer; 3) receiving CJMS materials 
with what they considered inadequate instructions as to their use and 
then not being able to obtain clarification in writing or through 
the personal appearance of a Technical Advisor. Staff members conceded 
that these experiences possibly had a negative effect on their opinion 
and ability to use CJMS. In one interviewing experience in this office, 
the office tnanager "suggested" that one counselor might be a very good 
person to interview, and this was a completely positive interview. 
The counselor responded to questions willingly and knowledgably, but 
gave the impression to the interviewer that each answer was being 
carefully thought out. This interview was conducted in the presence 
of another staff mentfier and was, at least in attitude, completely 
opposite the interview that had been conducted previously with this 
staff member. The person then became reticent about answering fur- 
ther questions. The atmosphere for conducting interviews in the 
rest of the office changed also, and further staff interviews were 
reduced to giving only direct answers to questions. Until that inter- 
view, staff members had spoken openly and freely. 

In Atlantic CUy, staff members spoke freely of CJMS but were reluc- 
tant to discuss the personality conflicts other than to acknowledge 
that they knew of the existence of these problems. The staff did 
mention difficulties in getting technical assistance with respect to 
CJMS until they had actually purchased the system at the end of the 
experiment. 

It is believed by Ultrasystems that the conflicts caused by these 
problems hindered the acceptance and effective use of CJMS by the 
Vocational Rehabilitation agencies. Why Atlantic City was not 
affected by these problems is unknown. What the Vocational Rehabili- 
tation agencies' experiences with and attitudes toward the systen 
would have been had these problems not existed is unknown. 
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THE JOB OUTLINE CHECKLISTS AND COST DATA 



2.5.1 Job_QtJtHne Checklists (JOCL'sl 

None of the conmunlty agencies or Vocational Rehabilitation agencies 
Involved In this study had a full-time job development staff. Job 
development was usually performed by Interviewers and counselors 
as a regular part of their services to clients. BICC alone, of 
all the agencies Involved In this study, prepared Job Outline Check- 
lists (JOCL's) for the jobs they harf developed. These JOCL's were 
entered Into the CJMS job bank and SICL's were usually matched 
against these JOa's. There were approximately two hundred JOCL's 
In BICC's job bank at the close of this experiment. 

Host of these JOCL's represented jobs with no known Immediate open- 
ings, but BICC preferred matching client SICL's to these JOCL's 
as opposed to the CJMS clusters used by all the other agencies, 
flugre 2-10 shows a sample page from the CJMS Job Cluster Register 
which was used In New Jersey as an aid to selecting appropriate 
occupations for clients. It was felt that strong SICL/JOCL matches 
had a good chance of being developed Into referrals and hires, even 
though the job was not known to be Immediately available. In addi- 
tion, there was some question as to the jobs represented by the 
clusters being directly comparable to-the jobs available In the 
Newark area. 

This question as to the applicability of clusters to Individual 
local labor markets was raised by all offices surveyed. The strong- 
est statement to this effect came from a Vocational Rehabilitation 
agency in which they pointed out that the job clusters were "developed 
from jobs in the south" and It Isn't known If the "behavior on jobs 
In this area Is the same as some other." A study with respect to 
validating the comparability of cluster jobs to local jobs was Iden- 
tified as being needed. 

As an alternative, It was suggested that the clusters be reshaped 
using JOCL's developed only from local jobs. This agency also 
stated that It had requested such a study from ADP-PDS and had not 
received a satisfactory response. Indeed, this appeared to be one 
of the blind spots for the developers of CJMS. In discussions with 
them, they mentioned what they regard as problems Inherent In the 
Dictionary of Occupational Titles used by the Employment Service as 
job definitions that are too broad. However, they could not recog- 
nize comparable problems In their own cluster data system. The 
CJMS developers further Insisted that their method of describing 
jobs in the clusters was adequate, yet Atlantic City counselors 
expressed a strong desire for additional definition, since they 
felt the behavioral par»neters used by CJMS were an Insufficient 
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description of job content. Figure 2-10 shows an example of one 
page taken from the cluster manual supplied by CJHS. It displays 
the dimensions of work for that cluster and a graph which provides 
a visual display of the dimensions. This Is followed by a listing 
of the jobs whose JOCL's are Included In that cluster. Each job 
Is identified by a job t1tle> Its subtitle, the business the job 
was found 1n> and a DOT code. Hot every job listed has a DOT code 
shown for It. According to the CJHS developers* they asked someone 
from the Employment Service to assign DOT codes to the jobs listed 
In the clusters and the jobs shown with DOT codes were the only 
jobs for which there were DOT codes. However, a quick review of 
the DOT book* shows DOT codes of 822.281 , 822.381 > and 8gg.887 
for Installer, Lineman, and Maintenance Helper, respectively. A 
single call to the Bnployment Service provided two possible codes 
for a Production Artist: 141.081 or 144.081. The former code 
refers to an "Art Layout Man" who "plans and arranges art layouts 
using sketches, pictures. . . The latter code refers to a 
"Painter" who "paljits landscapes, portraits, still life, abstract 
designs Aid similar compositions In oils, watercolor or tempera. . , 
This job title Is somewhat obscure, since it can. refer not only 
to the two types of artists shown above, but the latter artist 
(Painter) could also be working In mass production art. which Is a 
totally dlffej-ent worklnfi envlroranent from non-mass produced art. 



2.5.2 Costs 

BICC counselors felt that use of CJMS resulted In a significant 
aiiiount of time saved In the counseling process, since it reduced 
the time required for fact finding and helped the counselor set 
client service objectives fairly soon In the counseling process. 
While Atlantic City counselors did not notice a reduction In time 
required for counseling, some did think that CJMS helped them to 
set counseling objectives sooner than they might have without CJMS. 
TOPS counselors could offer no insight on this subject, since, as 
they explained to us, they "do not think In those terms." 

None of the experimental offices was set up to do cost studies with 
respect to a topic such as presented above. In fact, no cost studi 
were done. Me were told by the CJMS developers that It required 
approximately 40 minutes for an applicant to complete both booklets 
of the SICL. Administration times reported by each agency ranged 
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ANALYSIS OF THE PERFORMANCE DATA OBTAINED FROH 
THE NEW JERSEY/CLEFF EXPERIMENT 



As has been mentioned 1n earlier sections of this report, 
the New Jersey/Cleff experiment did not provide sufficient quanti- 
tative data for the evaluation of the CJHS as originally designed. 
This section will present the data that was obtained and the analy- 
S8S that were completed. The discussion will point out the prob- 
lems associated with the New Jersey data base. In addition, this 
section will present a critical review of the final report sub- 
mitted by AOP-POS* Inc. regarding the New Jersey experiment and 
Its results. The first subsection that follows will describe the 
planned and actually Implemented data collection activities associ- 
ated with the evaluation of the New Jers^/Cleff experiment. The 
next subsection will present the data obtained and analyses per- 
formed. This subsection will contain data and analyses associated 
with determining the socioeconomic characteristics of the control 
and experimental group clients, the referrals and placements of 
these client groups* and the post-placement results. The third 
subsection will draw upon the second In terms of reviewing the 
AOP-POS, Inc. final report. 



3.1 REVIEW OF THE SCOPE AND HETHUOOLOGY OF THE 

DATA COLLECTION 

A technical note, submitted by UUrasystems In August 1972* described 
In considerable detail the content and format of the data collected 
during the New Jersey experiment. The technical note proposed a new 
standardized data collection form to be used to collect the required 
data from both client groups and for all the agencies. This data 
collection form Is shown In Figure 3-1. The use of this form was 
agreed to by both OEO and the contractor running the New Jersey 
experiment, I.e., AOP-POS, Inc. OEO Instructed AOP-POS, Inc. to use 
this data collection form to record the data on the experimental and 
control group clients and to submit these coir^leted forms to UUrasystems. 
The data collection using this form began In the fall of 1972 and 
was completed In March of 1973. The date received on these forms 
was keypunched by UUrasystems and a data base for subsequent analy- 
sis was generated. This data base was created for use with the 
Statistical Package for the Social Sciences (hereinafter referred 
to as SPSS). 

Figure 3-2 shows the form of the data base defined for use by 
SPSS. The data base consists of a set of Individual cases, each 
of which contains the data pertaining to a given client. If one 
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examines either the data collection form or the data SPSS data 
base definition, one sees the following:' 

1) Each client Is identified by an assigned client identi- 
fication number and is also identified as to the agency 
with which the client was associated; the counselor In 
charge of this client's case file and the experimental 
or control group in which this client is Included. 

The client's latest status is also defined. There are 
seven status codes defined as follows: 

a) Pending Referral 

b) Pending Job Acceptance 

c) Pending Trainirig Acceptance 

d) Accepted Into Job 

e) Accepted Training 

f) Closed—Not Referred to Job or Training 

g) Closed— Referred to Job or Training But Not Accepted 

In addition, the date when the client first entered the 
agency is provided. 

This data is recorded on lines A, 8, and C. 

2) Each client is further identified by the client's social 
security number and if the client has been placed (accepted) 
into a job or training or is pending same, then the client's 
name, address, and phone number are provided. Clients who 
had been placed (or because they were in pending status could 
be placed) were to be interviewed. Therefore this data was 
required. 

This data is recorded on lines C through G. 

3) The socioeconomic characteristics of each client are pro- 
vided. These data elements are essentially the standard 
ones utilized in manpower evaluations. Because of the 
presence of Spanish-speaking and of vocational rehabllita- 

'tion clients in the data base, the data items listed In P 
and Q were included. In Q, the Vocational Rehabilitation 
Disability Codes are to be input. The client's individual 
and family Income at the time the client first entered the 
agency are given in item T. 



^ The line de;»ignat1ons used refer to the data collection form 
shown in Figure 3-1- 
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4) The client's work history is briefly provided in item R. 
Please note that in addition to the standard data items 
space is provided for the Inclusion of the Cleff Job Cluster 
register number that best represents each of the client's 
previous jobs. This item Is input based on the past job 
descriptions provided in the client's case files. It is 
thus a judgmental entry. 

5) The source of the client as viewed by the agency with which 
this client was associated 'during the experiment is Identi- 
fied in item S. In addition, item S provides data about 
the client's association with other agencies {if any), 
Including the other agencies participating in the experiment. 

This completes page 1 of the data collection form. As can be seen 
from the above descriptions, the data on page 1 is almost completely 
concerned with the client's characteristics and has no substantial 
data regarding the services and the outcomes of these services pro- 
vided by the agency. In addition, page 1 contains no data related 
to the client's CJMS scores. Page 2 of the data collection is de- 
voted exclusively to these data items. In particular, page 2 is 
devoted to obtaining the data that describes the referral services, 
their outcomes, and the CJMS scores as follows; 

1) The referrals to either jobs or training programs are 
described in item U. A cluster number is assigned based 
on the job description. This cluster number is taken 
from the Cleff Cluster Register and Is assigned based on 
the cluster register's list of jobs. If the job or train- 
ing program to which the client has been referred has been 
profiled usin^ the JOCL instriwent, then the number assigned 
to this JOCL IS included in the Job Title space. The JOCLs 
completed during the experiment were to be supplied Ultra- 
systems so that the correspondence could be made. 

The referral result is listed according to the codes listed 
on the form. 

2) If the client has been placed, then the employer or train- 
ing organization is Identified In block V. This data was 
needed for the interviews to be done. The date of hire 

and the date terminated or graduated {if applicable) Is also 
to be enter^. 

3) Follow-up data, if available, is entered in items W and X. 

4) The client's SICL dimension scores and the client's occu*- 
pational adjustment {factor) and difference indices are 
entered in block V. 
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The data obtained via this data collection fonn was to be supple- 
mented through interviews with clients placed into jobs and/or 
training programs and with their on-the-job supervisor. This data 
would provide the definitive data on retention and would also pro- 
vide measures of job satisfaction, job performance, and job advance- 
ment. As has been discussed earlier in this report, these inter- 
views were never conducted. Thus, the data provided Ultrasystems via 
the data collection instruments described above represents the only 
quantitative data available to Ultrasystems regarding the New Jersey/ 
Cleff experiment. 



3.2 ANALYSIS OF THE PERFORMANCE DATA OBTAINED 

Ultrasystems received data collection forms on 2619 experimental 
group clients and 1170 control group clients. These data collection 
forms were received from all the agencies participating in the New 
Jersey experiment with the exception of the Jersey City Vocational 
Rehabilitation office. In the technical note submitted by Ultrasystems 
In August 1972, it was pointed out that this particular Vocational Re- 
habilitation office had administered SICLs to only 18 clients through 
July 31, 1972. This particular office had essentially decided not 
to participate in the experiment. 

Ultrasystems proposed and OEO agreed to drop this agency from the 
experiment at least as far as obtaining data is concerned. The 
agency was visited and staff interviews were conducted. The results 
of these have been reported in the previous section. 

Table 3-1 shows the number of experimental and control group clients 
for which data collection forms were received. This table shows the 
number of such clients per agency and per the groups into which they 
were assigned. The experimental design called for three time-phased 
experimental groups and three time-phased control groups. The experi- 
mental design specified that the three e)q)erimental groups would each 
number approximately one-third of the total experimental clients 
to be processed, i.e., 3500 clients or 1167 per group. The first 
and third control groups would consist of 200 clients each, and the 
middle or second group would consist of 400 clients. Based on the 
analysis documented in the technical note submitted in August 1972, 
Ultrasystems recommended that the control group size be increased 
from the original 800. The intent was to try and equalize the sizes 
of the experimental and control groups for each agency except BICC and 
TOPS. The reasons why these two agency control groups were not 
included in this recommendation were as follows: 
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TABLE 3-1. NUMBER OF CLIENTS IN EACH EXPERIMENTAL AND CONTROL GROUP FROM EACH AGENCY; PER DATA RECEIVED 



Agency 


Experimental 


Con' 


trol 


Nar.e Number 


EI 




E3 


Total 


Cl 


C2 


C3 


Total 


Special C5 


Total Control 


BICC 1 


847 


370 


344 


1561 


99 


196 




295 




295 


Atlantic City MC 2 


55 


26 


58 


139 


15 


11 




26 




26 


Atlantic City VR 3 


134 


27 


22 


183 


15 


30 


229 


274 


22 


296 


Jersey City VR 4 






NO DATA AVAILABLE— DROPPED FROM DATA ANALYSIS 




New Brunswick VR 5 


111 


8 


6 


125 






33 


33 


93 


126 


Perth Amboy VR/MC 6 


130 


66 


37 


233 


38 


20 


125 


183 


42 


225 


TOPS 7 


137 




146 


283 


107 






107 




107 


FOCUS 8 




90 


5 


95 




73 




73 


22 


95 


Totals 


1414 


587 


618 


2619 


274 


330 


387 


991 


179 


1170 
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(Note: In the ADP-PDS, Inc. final report to be discussed 
In the next section* the data about BICC» TOPS, and FOCUS 
were presented uniquely. The Perth Amboy and Atlantic City 
Hodel Cities (agencies HZ and 1II6) data were grouped under 
the heading Model Cities. The data from the Atlantic City 
VR and New Brunswick VR offices (agencies 3 and 5) were grouped 
under the heading VR). 

It is clear, however, that the experiment did not comply 
with its original design regarding the number of clients to 
be processed by the three agency groupings. 

4) In the description of the experiment that accompanied the 
RFP for the evaluation* it was stated that each of the three 
time-phas^ed experimental groups "will consist of every agency 
applicant meeting both agency criteria and CJMS criteria for 
placement consideration until one-third of the particular 
agency quota has been met (see Appendix D). Ultrasystems has 
never seen any document that sets forth these agency quotas. 
In addition the experimental and control groups were to be 
time-phased. The key here was that the CJMS would be used 
and not used over distinct time periods. When the CJHS 
was not in use, the applicants meeting each agency's criteria 
for placonent consideration and being accepted by the agency 
as a client would be included in the control groups. The 
experimental design naver, to Ultrasystems' knowledge, addressed 
the relationship between agency quotas and the timing of the 
groups. In addition, it never apparently addressed the issue 
of the comparative experimental' and control group demographic 
similarity in terras of the decision regarding inclusion 
of clients Into the control groups. Some of the issues Involved 
here are: 

a) If each experimental phase began at the same time for 

each agency* then It Is conceivable, and in all probability 
likely* that each agency would reach its quota at different 
points in time. If this happened, would one wait until 
all the agencies meet their quotas (or were close) before 
one started including clients in the control group? Then* 
as with the experimental group* one could conceivably have 
the r.ontro1 group quotas met at different times. 

b) One could have begun the experiment at the same time but 
have each agency transition between experimental and control 
groups when they meet their quotas. This would yield a 
continuity to each agency's use and non-use but could lead 
to differences In the absolute times at which each agency 
was In one of the two operating modes. 
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c) One could have a priori determined chronological time 
points for the use and non-use of the CJMS by all the 
agencies. This could result in agencies exceeding or 
not reaching their quotas, but it would have yielded dls* 
tinct time phases in which the system was used or not 
used. In addition, there would be continuity over time. 

d) The description of the experiment referred to earlier 
(see Appendix D) Indicated that when the experiment was 

In the control group phase {CJMS was "turned off"), all the 
clients accepted by each agency (meeting the agency criteria 
and the CJMS criteria for placejnent consideration) would be 
included in the control group. The description also used 
the phrase "time dependent control groups." Thus, It seems 
that the experimental design was calling for distinct time 
periods during which the CJMS would be used and not used 
and that all clients meeting a set criteria that came to 
each agency durinn the time periods would end up in the 
appropriate group. The concept of quotas might be more 
related to the choice of the time spans based on estimates 
of agency volumes and need not be rigidly adhered to during 
the course of the experiment. 

In the final analysis, the experiment did not proceed according to 
these design concepts. One agency, BICC, used the CJMS essentially 
all the time for all Its clients. This agency's control group was 
comprised of clients served by another agency. The TOPS agency used 
the CJMS for all the clients it was screening for acceptance Into 
Its training programs. The control group for TOPS consisted of clients 
accepted into training prior to the experiment. TOPS does not operate 
with a continuous Intake process, but rather screens applicants for 
acceptance into Its tralnlnp at distinct time points. Some of Its 
applicants are screened, but are automatically accepted because 
of arrangements with uther agencies who refer clients to them. All 
the other agencies or offices used the CJMS essentially continuously, 
but did not use it for all clients over any time period. The control 
groups for these agencies were obtained by som sort of random case 
file selection process. The experiment, therefore, did not operate 
In distinct on-off time periods and did not include all clients 
over some time periods who met some criteria into either the experi- 
mental or control groups. In all fairness to the contractor running 
the experiment. It should be noted that little. If any, control was 
exercisable over the agency's operations during the experiment. In 
Ultrasystems' opinion. It would not be reasonable or even desirable 
to have such control exercisable by an outside contractor over an 
agency's operations. For all practical purposes, the original design 
was probably unrealistic. 
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Because of these aspects, the data to be presented treats the experi- 
mental and control groups as distinct entities with no breakouts given 
in terms of time-phased groups. The data about the experimental and 
control groups are, however, discussed in terms of their agency asso- 
ciation. 



3.2.1 Demographic Characteristics of the Experimental and 
Control Group Clients 



Tables 3-2 through 3-8 present the demographic characteristics of the 
experimental and control group clients from each agency. The tables 
are all in the same format. 

Examining Table 3-2, as an example, one sees that the demographic data 
is shown separately for the total experimental and total control group 
clients for the agency. Each demographic data item is shown in terms 
of the observed frequency with which it was reported. The frequency 
counts for each data item do not contain counts of missing or unreported 
data. Thus many of the frequency counts shown do not add to the total 
number of clients reported in each group from each agency (see Table 3-1), 
The percent of the observed frequency for each data item is also shown. 
The chi-square statistic comparing the frequency count betweenthe ex- 
perimental and control group for each attribute data item is also shown 
along with a symbol notation that indicates the significance of the chi- 
square statistic. The symbol notation is interpreted as follows: 

- Not significant at lowest confidence level at which 
tests were made (i.e., 955S) 

+ Significant at 95X confidence level 

++ Significant at 99X confidence level 

+++ Significant at 99,9% confidence level 

The variable "highest grade" is shown at the bottom of the table. This 
was treated as a continuous variable and the t-test was used to compare 
the means of the two client groups. The symbol notation given above is 
the same. 

The statistical measures are read in terms of the null hypothesis; i.e,, 
that the relative frequencies of the demographic characteristic (attribute) 
shown are the same between the experimental (CJMS) and control groups. 
Hence the presence of one or more + symbols indicates that the null hypo- 
thesis does not hold at the symbol-indicated level of confidence. 
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TABLE 3-2. ANALYSIS OF DEMOGRAPHIC DATA FROM THE NEW JERSEY/CLEFF EXPERIMENT 

jtfiOlC T n - BICC 



CHftRAaERISTICS 




CONTROL Group 




OBSERVED FREOIEICV 




OBSERVED FREQUENCY 1 11L\ 


X* 


SIG 


sex (X male) 


426/1559 


27.3 


1 1 0/ 


! ^7 ^ 


11 .50 


+++ 


Ethnic 
White 


72/1408 


5.1 


43/25S 


16.7 


43.51 


+++ 


Black 


1266/1408 


M.9_ 




b\ .0 




+*+ 


Spanish 


si/Mca 


— 3-8— 


6/252 


2.3 




- 


Criminal Record 


137/1550 


8.8 


NO udra 




- 




Speak English 


1421/1430 


99.4 


36/36 


100.0 


0.36 




Read EnQllsh 


1417/ld.'*n 


qq.l 


35/35 


inn.o 


n.i? 


- 


Write Ennllsh 


1409/1430 




3Q/3Q 


inn.n 


n.m 




Citizenship 


1304/1354 


96.3 _ 


57/57 


mL.0 


1 .24 




Drive Car 


f^l Q /I ceo 




83/220 


37.7 


n OQ 

U. ^7 




Own Car 


325/1 548 


Z1 .0 


55/209 


26. 3_ 


2.77 




Barriers to Employment 
Lack of Skills 


2/1561 


0.1 






b.oo 




Dependent children 




7 9 


^ # Art ^ 

1 /?Q<i 


— 0_3_ 


iQ in 


+ + + 


Lack of Experience 


I2a/1561 




5/295 


1 -7 






Language - 


7/1561 


0.4 


U/ CUD 




0.40 


- 


Too Old 


32/1561 


2.0 


1/295 


0.3 


3.24 




Ex-Offender 


69/1 561 


4.4 


0/295 


0 


12.34 


+++ 


Physical Handicap 


8/1561 


0.5 


0/295 


0 


0.56 




Lack of Education 


543/1 561 


34.8 


33/295 


11.2 


63.46 


+++ 


Mental Handicap 


-in (;fii 


n? 




n 


n.nm 
























1 







VARIABLE 



EXPERIMENTAL SRQUP 



COHTRO 



6R0UP 



SIG 



Highest Srad< 



11 .6/ 1 KQ/! 



1 97 



5 as 



at 



U3 
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TABLE 3-3. ANALYSIS OF DEMOGRAPHIC DATA FROM THE HEW JERSEY/CLEFF EXPERIMENT 

MENCr n Atlantic Ci ty Modal C1t1« 



VARIABLE 
CHARACTERISTICS 



Highest Grade 



ElfPERrwEKTAL GftOUP 



T 



inno 



2. S3 



COHTgQi SROUP 



10^00 



2,4 9 



ITTRIAitYg . - - — - 

CHARACVERISTICS 


tXPERlHEKTAL GROUP 


COKTROL GROUP 


X2 


SIG 


OSSEftVED FftEOUEHCt 




OBSERVED FREQUENCY 




Sex (x male) 


74/139 


53,2 


lS/25 


f;7.7 







Ethnic 
White 


54/138 


38.8 


9/26 


34-6 


0.04 






83/138 


59.7 


17/26 


65.4 


0,08 




Spanish 


1/138 


0,7 


0/26 


0.0 


0,88 




Criminal Record 


30/136 


22.1 


5/26 


19,2 


0-02 




Speak English 


128/132 


97.0 


25/25 


JOD.CL 


Q.Q4 




React English 






?t;/9»; 


ino.n 


0,04 




Write English 


i?a/i^? 


Q7.n 


?R/9R 


mn n 


0.04 


1 


Citizenship 


11 ^ /1 1 1 

112/112 


100,0 


?fi/?fi 


100.0 


n.on 




Urive Car 


13/104 


12.5 


9/25 


36-0 


5,75 


+ 


Own Car 


9/1 04 


8,7 


2/25 


8.0 


0.09 




Barriers to Employment 
Uck Of Skills 


0/1 39 


0,0 


0/26 


0,0 


0,00 


- 


Dependent Children 


0/139 


0,0 


0/26 


0.0 


0.00 


- 


Lack of Experience 


7/139 


5.0 


0/26 


0.0 


0.41 


- 


Language 


0/139 


0.0 


0/?fi 


0.0 


n.oo 




Too Old 


7/139 


5.0 


0/26 


0-0 


0.41 


- 


Ex-Offender 


7/139 


5,0 


1/26 


3.8 


0.06 




Physical Handicap 


S/I^q 


5.S 


0/?fi 


0.0 


0.R7 




Lack of Education 


_ -_a&/i2Q_ 


61.2 


11/26 


42_3 


2.47 




Hental Handicap 


16/139 


11. S 


0/26 


0.0 


2-13 

































SIG 



TABLE 3-4. ANALYSIS OF OEMOGRAPHIC OATA FROM THE NEW JERSEY/CLEFF EXPER!MEI>fT 



Vocational 
AGtWC r »3fttUnt1c City -Rehabilitation 



CmMCTEKlSriCS 


OfEttNEffTAL CMNIf 


COmTROL wOU? 


X 
















ob*u 




53.0 


6.10. 


+ 


Ethnic 
Unite 


112/17fi 


OtJ * o 


/ O/ 1 ^ 1 


C9 ft 


0.001 




Black 


90/ I/O 


^1 Q 
J WO 


AA/1 Ol 




0.010 


^ I H 1 


Spanish 


^/ 1 /o 


1 1 
1*1 


C /I Ol 


>i 1 
4.1 


1.65 




Criminal Record 


55/1/8 




OO /OQQ 
CC/CQO 


/ ./ 


44.48 


+++ 


Speak English 


180/180 


^ nA 

100 


289/289 


100 


n.oo 




tt«ad Enqllsh 


17q/17q 1 




2ftQ/2ftQ 


\m 


n.nn 




Write Enqllsh 


179/179 


ino 


?fi9/?ftQ 


inn 


n.nn 




Citizenship 


179/179. 


100 


271 /282 


96.1 


5. 58 




Drive Car 


n/177 




43/27? 


1R.8 


ft. 44 


+ + 


Own Car 


9/177 


5.1 


22/268 


8.2 


1.16 




Barriers to E*n'i>loyment 
Lack of Skills 


1/183 


.5 


1 04/296 


35.1 


77 OA 




Dependent Children 






A /OAC 




n.nn 




Lack of Experience 




1 1 




a 1 


11 .44 


+++ 


Lanqtiage ' 


U/ 




A/OAC 


n 


n.oo 




Too Old 


6/I8J 


3.3 






n.n4 




Ex-Offender 




Lt-! 


J/ C^2U ^ 




1 .14 




rnysica i nanoicap 


20/133 


10.9 


127/296 


42.9 






Lack of Education 


88/183 


48.1 


167/296 


56.4 


2.83 




Hental Handlcao 


36/183 


19.7 


63/294 


21.4 


0,12 

































VARIABLE 
CWRACTERISTICS 



EXPERIMEHTAL GROUP 



COHTRO 



6R0OP 



SI6 



Highest Grade 



10.03 I 2.50 
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TABLE 3-5. ANALYSIS OF DEMOGRAPHIC DATA FROM THE NEW JERSEY/CLEFF EXPERIMENT 

AfiEMCy #5 



Vocational 
New Brunswick -Rehabilitation 



CHARACTERISTIC^! 






X2 


SI6 


_085ERVE0 FREQUEHCY 




OBSERVED FREgUEHCY 


1%) 


Sex {% malft) 








DO •/ 


7.03 - 


++ 


Ethnic 

mi 1 w 


50/119 


42.0 


27/40 


67.5 


6.80 


++ 


OldCK 


67/119 


56.3 


10/40 


25.0 








2/119 


1 .7 


3/40 


7 5 


\ .by 




i*riinind i necora 


21 /ni; 


in 2 


n/d^ 


25.6 


DBA 




Speak Enqllsh 


77/77 


mn 




inn 


Q-no 




Read English 


77777 


100 


_ 37/38 


97.4_j 


0.13 




Write Enqllsh 


77/77 


100 


37/38 


97.4 


0.13 


- 


Citizenship 


R.'i/53 


TOO 


12/12 


100 


0.00 




Drive Car 


71/117 


60.7 


1 S/3q 


38.5 




+ 


Own ICar, 




36. fl 




2ft.fi 


? 4? 


_ 


Barriers to Eiiio1ovni#n^ 
lack of Skills 


1/125 


0.8 


0/126 


0.0 


0.00 




Dependent Children 


4/l?R 


3-2 


5/126 


4.0 


0 00 




Lack of Experience 


6/125 


4.8 


8/126 


6.3 


0 07 




Lan^uaqe 


D/125 


O.Jl_ 


1/126 




fl 00 




Too Old 


9/125 


7.2 


18/126 


14.3 


2.59 




Ex-Offender 


4/125 


3.2 


11/1 26 


8.7 


2.50 




Physical Handicap 


36/125 


28.8 


1/126 


0.8 


36.96 


+++ 


Lack of Education 


6fi/125 


54.4 


55/126 


43.7 1 


2.49 




Hental Handicap 


6/125 


4.8 


23/126 


18.3 


9.84 


++ 































EXPERII 


HEHTAL GROUP 




COMTRO 


L GROUP 


VARIABLE 
SHftRACTERISTICS 


T 


S 


n 




X 


s 


n 


t 


SIG 


Hiqh«St Grade 


10.55 


2.37 


125 




in. 21 


?.afi 


HQ 


i.m 

























TA8LE 3-6. ANALYSIS OF DEMOGRAPHIC DATA FROM THE NEW JERSEY/CLEFF EXPERIHEKT 



MENCy #6 Perth A«bov >tod#1 CitiM 



00 



CKHmCTERISTICS 


IS! 


QKItmOITM. OtOUP 


COifTMt aoup 


SIG 




V*/ 


OBSntVED FREOUENCY 


VMl 


X* 


Sex (X male) 


157/233 


07.4 


137/223 


61 .4 


1 .51. 


- 


Ethnic 
UMte 


46/232 


19.8 


33/209 


15.8 


.96 




Black 


30/232 




LAJ cuy 


in n 




** 


Spanish 


155/232' 


DO.S 










Criminal Record 


29/173 






Z6.4 




- 


Speak English 


120/188 




7 74/166 


68-7 




- 


Read English 


109/181 


60.3 


92/157 


58.6 


.04 




Write English 


105/179 


58.6 


89/1 5S 


56.3 


.10 




Citizenship 


125/142 


88.0 


72/96 


75.0 


5.93 




Urivt Car 


75/211 


35, S 


26/54 


48,1 


2.39 




Own Car 


42/216 




8/40 


20.0 


.02 




Barriers to EmPlaynwnt 
Lack of Skills 


11/233 


4.7 


1/225 


0 4 


6 61 


+ 


Dependent Children 


?/?.r3 


n 0 


— Lli/ ^/h 


A A 


A At; 




Lack of Experience 


8/233 


S^3_ 










LanSuaQe ' 


49/233 


?^ 0 


33/225 


14.7 


1ft sn 


+++ 


Too Old 


n/233 


d 7 




6 7 


— 




Ex-Offender 


16/233 






8 0 


nft 


** 


Physical Handicap 


48/233 


20.6 


78/225 


34.7 


10.66 




Lack of Education 


141/??13 


60.5 






11 90 


+++ 


Mental Handicap 


14/233 


fin 




1.8 




+ 






























EXPERI 


MENTAL GROUP CQHTRO 


L GROUP 


VAftlASLE 
CHARACTERISTICS 


J 


s 


n 




J 


s 


n 


t 




Highest Grade 




2.61 






R fie. 


3.46 


:213 I 






















1 





TABLE 3-7- ANALYSIS OF DEMOGRAPHIC DATA FROM THE NEW JERSEY/CLEFF EXPERIMENT 

MtMC Y #7 ' TOPS 



CHWACTESISTICS 


OUQtVa FREQUENCY 


Ix) 




X2 


SI6 


Sex (X male) 


39/279 


14.0 


3/ lUO 




5.63. 


+ 


Ethnic 
UMte 


52/276 


18.8 


16/103 


15,5 


.36 


- 


Black 


19/?7fi 


fi.q 


Til illA 








Spanish 


188/276 


68.1 


/a/ lUJ 




.89 


_ 


Criminal R«:ord 


21/l?fi 


Ifi.d 


Rad Data 








Speak English 


201/247 


8K3 


52/58 


89.7 


1.73 




Read £n<}11sh 


196/247 


7?t4 


51/57 


59.5 


2.49 




tfrlte English 


195/247 


79J) 


50/_57 


87.7 


1-75 




CItlzenshiD 


195/265 


73.6 


52/60 


86.7 


3.90 


+ 


Urive Car 


17/1. -11 


13.0 


Bad ' Data 









Own Car 




12.2 


Bad Data 








Barriers to EMplo^nt 
Lack of Skills 


36/283 


12.7 


19/107 


17.8 


V.24 




Dependent Children 


10/283 


3.5 


7/107 


6.5 


1.04 ■ 




Lack of Experience 


lfi/9M 


5.7 


10/107 




l.lfi 




language ^ 


60/2B3 


2K2 


13/107 


12.1 


3.61 


- 


Too Old 


11/283 


3.9 


5/107 


4.7 


.004 




Ex-Offender 


0/283 


0.0 


0/107 


0.0 


.00 




Physical Handicap 


1/283 


0.4 


2/107 


1.9 


.77 




Lack of Education 


123/128 


43.5 


23/107 


21.5 


15.07 




Hental Handicap 




0.0 


0/107 


Q.Q 


0 

































VakIaBLE 
CWRACTfaiSTICS 



EXPgRlHEHTAL GROUP 



COHTRO 



CROUP 



sm 



10.86 1.78 280 



10.35 



2.43 



106 2.26 
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TABLE 3-8. ANALYSIS OF DEMOGRAPHIC DATA FROM THE NEW JERSEY/CLEFF EXPERIMENT 



STTrTSvt? " 
CHARACTERISTICS 



OBSERVED FREQOEWCY 



cowtrol gikxjp 



OBSERVED FREQUENCY 



HI 



SIG 



Sex (t male) 



48/95 



50.5 



56/95 



58.9 



1.04 



Ethnic 
White 



0/95 



0.0 



2/95 



2.1 



0.51 



Black 



Spanis h 



0/95 



(LQ_ 



■^5 



7.46 



++ 



Criminal Record 



Speak English 



7/91 



15/74 



Jim. 



2£L3. 



JUL 



AAJL 



5.42 



0 



Read English 



1f>/7T 



5-?1 



Write English 



3R.7 



Citizenship 



48/73 



65.8 



66/87 



1.52 



Drive Car 



Own Car 



44/04 



24/93 



46.8 



41 /95 



■25.8 



27/81 



43.2 



0^13 



33.3 



0.85 



Barriers to Bnployment 
tack of Skills 



1/95 



1.1 



0/95 



0.0 



0.00 



Dependent Children 



0/95 



0.0 



0/95 



0.0 



0.00 



Lack of Experience 



n/95 



n-6 



1/95 



7.21 



Language • 



Jim. 



49.5 



50/95 



52.6 



0.08 



Too Did 



Ex-Df fender 



Q/9S 



1/95 



Jim. 



1.1 



0/95 



0.0 



0.00 



0.0 



0.00 



Physical Handicap 



0/95 



Q.O 



0/95 



0.0 



0.00 



Lack of Education 



Mental Handicap 



65/95 



0/95 



68.4 



60/95 



JUL 



0/95 



63.2 



0.37 



JLSL 



0.00 



EXPERIMENTAL 6 


ROUP 




CDHTRO 


L GROUP 


.•ARtABLE 
CHARACTERISTICS 


Jf j s - 


n 1 






s 


n 


t 


SIG 


Highest Grade 


9.73 12.35 


.94_ ! 




9.59 


2.77 


95 


0.37 






1 














1 



f 



The demographic data analyzed and presented in these tables is a 
subset of the data obtained from the data collection forms shown 
earlier, When it became apparent that the analysis of the New 
Jersey experimental data would not be sufficient for the purpose 
of evaluating the CJMS, the continued analysis of these demographic 
data was halted. 

The data presented in Tables 3-2 through 3-8 show that for each agency 
there was at least one demographic characteristic which was signifi- 
cantly different between the two groups. As an example* the data from 
8ICC (Table 3-2) shows that there were eight demographic variables 
significantly different at the 99.9% level of confidence. This in- 
cludes the sext ethnic* and education characteristics. 



3.2.2 Referral, Placement and Follow-up Data 

Tables 3-9 through 3-16 sumnarize some of the data obtained from the 
data collection form, shown earlier, regarding the job and training 
referrals recorded for the experimental and control groups. All the 
tables are in the same format, Table 3-9 contains the total data 
across all the agencies. The remainder of the tables show the data 
on an individual agency basis. 

Almost all the data shown in these tables come directly from the dai'a 
collection form shown in Figure 3-1. The tables have been divided 
into four blocks with each block numbered. The correspondence between 
the data collection forms and the data in each block will be briefly 
discussed before the data itself is analyzed. 



Block ^vl^. The data showing the distribution of total experimental 
and control group clients by their status at the time 



the data were collected comes from the "Client's Current 
Status" entry on line 8 of page 1. The data collection 
form listed 7 legitimate code values. The block®list 
contains two other code values which were only to be 
used to record referv<al outcomes on page 2. These two 
values, i.e.. Not Accepted by Employer and Not Accepted 
by Trainer, appeared often enough In "Client's Current 
Status" to warrant them being listed. They are, for the 
purpose of ascertaining the client's current status, 
equivalent to Closed-Referred-Not Accepted. 




11-65 



ERIC 



181 



^ 00 

to 



t 



© 



TABLE 3-9. REFERRAL, PLACEMEKT'ANO FOLLOW-UP DATA FROM THE NEW JERSEY/CLEFF 
EXPERIMENT 





Experimental 


Control 


Totsl Processed 


2619 


1170 


Total Referred 


1186 (45.3) 


411 (35.1) 


Client Status- 
Pt:^ "ij Referral 


237 ( 9.0} 


2 ( 0.2} 


Pending Jot^ Acceptance 


12 ( 0.4) 


1 f 0.1) 


Pending Training Accept. 


8 ( 0.3) 


1 ( 0.1) 


Accept Into Jot^ 


455 (17.4) 


206 (17.6) 


Accept Training 


215 ( 8.2) 


36 ( 3.1) 


Closed Hot deferred 


1194 (45.6) 


757 (64.7) 


Not Accepted by Einployer 


4 f 0.21 




Hot Accept by Train. 


31 ( 1.21 . 


0 (90 


Closed'Rfd^rtot Accept. 


(17.6) 


le? (14.3) 


Out of Range 


2 ( 0.1) 


0 0 



Referral Result Codes 





Pend. 
Ref, 


Pend. 
Job 

Accept . 


pend. 
Tmg. 
Accept . 


Accc 
Job 


ipted 
TrainifW 


1 B, 


Ho 

By 

Train. 


t Accepted 
By CI 
Valid . 


lent 
NonV. 


Unknown 


Hissing 
i Out of 
Ranqe 


Ref err 

Job 


•»} Code 
Training 


total 
ttos, of 
Ref. 


Humber 
of Ref. 
iMtes 


Exp. 
8ef. 
Result: 
































1 


0 


12 


9 


445 


346 


172 


55 


16 


53 


48 


1463 


703 


457 


1160 


1142 


2 


1 


3 


0 


58 


16 


54 


10 


6 


10 


10 


2451 


138 


31 


169 


166 


3 


0 


1 


0 


2 


2 


13 


0 


0 


2 


* 1 


2598 


20 


2 


22 


21 


4 


0 


0 


0 


0 


0 


3 


1 


0 


0 


0 


2615 


3 


1 


4 


4 


Control 

Ref. 

Itcsult: 
































-1 


1 


0 


4 


197 


41 


95 


30 


10 




12 


779 


3Q8 






llftQ 


2 


0 


0 


* 0 


18 


1 


14 


0 


1 


1 


0 


1135 


33 


2 


35 


35 




0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


1167 


3 


1 


4 


4 




0 


1 im- 


D i 0 


0 


0 


0 


0 


0 


0 


1170 


0 


d 


0 


0 



TABLE 3-9, REFERRAL. PLACEMENT ANO FOLLOW-UP OATA FROM THE NEW JERSEY/CLEFF EXPERIMENT 
ASENCy Total S 



DATA ABOUT REFERRALS 

CD 



Experimental 


Cluster" 
Number 


Date 
Hired 


Date 
Tenninated 


Grav Trvining 
Yes e No 


Date 
Grad> 


Ref. 1 


597 


30 


11 


8 


5 


9 


2 


85 


4 


1 


0 


0 


0 


3 


12 


0 


0 


0 


0 


0 


4 


1 


0 


0 


0 


0 




Control 














Ref . 1 


210 


50 


5 


2 


0 


6 


Z 


17 


10 


0 


1 


0 


1 


3 


2 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 



© 





Still 
Yes 


Job f o1 
Working 
No 


I lowup 
Date of 
Fol lowup 


Still in 1 
Yes 


Training F 
'raining 
1 No 


o1'!owup 
Training 
Yfs 


Grad. 
No 


Date of 
Fol 1 owup 


Training 
Grad, Date 


Experimental , 
Fo]lowup 


202 


62 


276 


106 


201 


80 


207 


309 


79 


Z 


2 


2 


2 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control 
Fol lowup 


64 


18 


82 


5 


10 


2 


9 


15 


5 


2 


2 


1 


3 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 



«€HCy # 1 BICC 



'Table '3-To/ 

EXPERIMENT 



■RE7fRRAL, PUCEMENT~MD FOLLW-UP MTA "FROM TH^ NEW" JErSEY/CLEFF 



t 

o 





Experimental 


Control 


ToUl Processed 


1561 


295 


Total Referred 


599 (38.4) 


113 (33.3) 


Client Status: 
Pending Referral 


147 ( 9.4) 


0 


Pending Job Acceptance 


8 ( 0.5) 


0 


Pending Training Accept. 


3 ( 0.2) 


0 


Accept Into Job 


186 (11.9) 


39 (13.2) 


Accept Training 


71 ( 4.5) 


0 


Closed not Referred 


814 (52.1) 


182 (61.7) 


Not Accepted by Employer 


1 ( 0.1 ) 


0 


Hot Accept by Train. 


1 ( 0.1) 


0 


C1osed"Rfd-lfot Accept. 


329 (2K1) 


74 (25.1) 


Out of Range 


1 ( D.l) 


0 



Referral Result Codes 





Pend. 
Ref. 


Pend. 
Job 

Accept. 


pend. 
Trrg. 
Accept. 


Accc 
Job 


Tr»lntr» 


By 


Trtln. 


t Accepted 
By CI 
VtHd . 


ient 
ttonV. 


Unknom 


Hissing 
( Out of 
Rsnge 


Referr 
Job 


^^ code 
Trtlntno 


ToUl 
Hos. of 
«ef. 


Number 
of Ref. 
iMtcs 


&p. 
Re/. 
Resultr 
































1 


0 


8 


4 


173 


130 


150 


31 


7 


44 


41 


973 


390 


201 


591 


562 


2 


1 


3 


0 


36 


10 


42 


10 


0 


7 


5 


1447 


90 


25 


115 


112 


3 


0 


1 


0 


1 


2 


6 


0 


0 


0 


1 


1550 


10 


2 


12 


11 


4 


0 


0 


0 


0 


0 


1 


1 


0 


0 


0 


1559 


1 


1 


2 


2 


Control 

Ref. 

Result: 
































.1 


0 


0 


0 


38 


0 


61 


5 


0 


6 




185 


104 


^ 


109 


119 


2 


0 


0 


* 0 


2 


0 


8 


0 


0 


0 


0 


285 


10 


0 


10 


10 


3 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


293 


2 


0 


2 


2 


AC ^ 


0 


0 


0 


0 


0 


0 


. 0 


0 


0 


0 


295 


0 


0 


0 


0 



REFERRAL, PLACEMENT AND FOLLOW-UP OATA FROM THE NEW JERSEY/CLEFF 



DATA ABOUT REFERRALS 



EXPE 


RIMENT 


Experimental 


Cluster' 
Number 


Date 
Hired 


Date 
Fermi nated 


Grad Tr 
ires 


aining 
NO 


Date 
Grad. 


Ref. 1 


268 


24 


5 


8 


5 


9 


2 


58 


1 


0 . 


0 


0 


0 


3 


6 


0 


0 


0 


0 


0 


4 


1 


0 


0 


0 


0 


0 


Control 














Ref. 1 


12 


5 


0 


0 


0 


0 


2 


1 


2 


0 


0 


0 


0 


3 


1 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 





Still 
ires 


Oob Fol 
voi'king 
No 


llowup 
Date of 
Followup 


Still in 1 
ires 


Training F 
raining 
No 


ollowup 
Training 
ires 


Grad. 
1 No 


Date of 

irollowup 


Training 
Grad .Date 


Experimental , 
Fol lowup 


32 


31 


63 


13 


97 


43 


61 


108 


36 


2 


0 


2 


2 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control 1 
Fol lowup 


0 


3 


3 


0 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 



AGENCY I 



T~Ayrantic Cl tv TABLE 3-TC REFERRAL, PLACEMENT 'ASo'TOLLdW-lIP DATA FROM THE"NEW JERSEY/' 
Model Cities CLEFF EXPERIMENT 



^ CO 



".4 

o 





ExperliMfital 


Control 


Total Processed 


139 


26 


Total Referred 


43 (30,9) 


11 (42 3) 


Client Status: 
Pending Referral 


33 (23.7) 


0 


Pemlln^ Job Acceptance 


0 


0 


Pending Training Accept, 


0 


0 


Accept Into Job 


17 fl2.2^ 




Accept Training 


21 (15.1) 


0 


Closed Kot Iteferred 


63 (45.3) 


15 (57.7) 


Hot Accepted by Employer 


0 


0 


Hot Accept by Train, 


0 


0 


C1osed-ltfd-l4ot Accept, 


5 ( 3.6) 


0 


Out of Itartge 


0 


0 



Referral Result Codes 





! Accept. 


Pend, j 
Trag. ' 
Accept. . 


1 Acc( 
[ Job 


tpted 
Tralnina. 


By 


Mo 

By 
Train. 


t Accepted 
By C] 
Valid 


lent 
IHonV. 


Unknown 


Hisslns 

t 0"' -f 

' ^ 1 


(tefert 
Job 


'«1 Codt 
Training 


Total 
NoSt of 
Ref. 


HuMbcr 
of Rtf. 
tfttes 


Exp. 
R*f. 
Resul t: 




1 




























1 


0 


0 


0 


15 


27 


0 


0 


0 


0 


0 


97 


16 


27 


43 


43 


2 


0^ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


133 


1 


0 


1 


1 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


139 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


139 


0 


0 


0 


0 


Control 

Jtef. 

RtsuU: 


































0 


0 


0 


11 


1 


0 


0 


n 


n 


rt 


14 


9 


1 


in 


4 


2 


0 


0 


/ 0 


0 


0 


0 


0 


0 




0 


26 


0 


0 




n 


3 


0 






0 


0 


0 


0 






0 


26 


0 


0 


0 


0 




0 


0 . 


0 


0 


0 


0 


0 


0 


0 


0 


26 


a 


0 


0 


0 



AGE«CYjf2_fitl antic City Model 



TABLE 
3-11. 



REFERRAL. PLACEMENT AND FOLLOW-UP DATA FROM THE NEW 



DATA ABOUT REFE 



t 



Experimental 


Cluster' 
Number 


Date 
Hired 


Date 
rerminated 


Grad Tr 
Yes 


aining 
No 


Date 
Grad. 


Ref. 1 


7 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


Control 














Ref. 1 


0 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


3 


n 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 




0 





Still 
Yes 


Job Fol 
Working 
No 


tlowup 
Date of 
Follovntp 


still In 1 
Yes 


Training F 
raining 

No 


ollowup 
Training 
Yes 


Grad. 
No 


Date of 
Follovntp 


Training 
Grad. Date 


Experimental . 
Fouowup 


1 


0 


1 


5 


8 


7 


6 


' 13 


6 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control f 
FollowuP 


0 


0 


0 


0 


1 


1 


0 


1 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 



o 

ERIC 



WEiCY I 3 Atlantic Citv Vocational TA8LE 3-12. REFERRAL. PLACEMENT AND FOLLOW-UP" DATA FROM " 
Rehabilitation THE NEW JERSEY/CLEFF EXPERIMENT 



CD 

i 00 





Experimental 


Control 


Total Processed 


183 


296 


Total Referred 


87 


(47.5) 


44 


(14.9) 


Client Status: 
Pending Referral 


17 


( 9.3) 


0 


Pending Job Acceptance 


0 


0 


Fendinj Training Accept. 


0 


0 


Accept Into Job 


36 


fl9.7) 


33 


ni.i) 


Accept Training 


34 


(18.6) 


7 


( 2.4) 


Closed Hot Referred 


78 


(42.6) 


252 


(85.1) 


Not Aiicepted by Qi^loyer 


0 


0 


Kot Accept by Train. 


0 


0 


Closol-Rfd-Hot Accept. 


17 


( 9.3) 


4 


( 1.4) 


Oui of 'Range 


1 


( 0.5) 


0 



Referral Result Codes 





Fend. 
Rtf. 


Pend. 
Job 

Accept. 


Pend. 
Trng. 
Acc«>t. 


Accf 
Job 


!Pte4 
TralniM 


By 

Dnpl. 


Ho 

Train. 


t Acc^ted 
By CI 
Valid 


lent 
NonV. 


UnknoMn 


Hissing 
it Out of 
Sanae 


Refert 
Job 


"at Code 
Training 


Total 
Mos. of 
Kef. 


HuiAer 
of Ref. 
tMtcs 


Exp. 
Kef. 
ttesuU: 
































1 


0 


0 


0 


34 


42 


2 


3 


1 


3 


0 


93 


38 


45 


83 


86 


2 


0 


0 


0 


4 


0 


0 


0 


0 


1 


0 


178 


5 


0 


5 


5 


3 


0 


D 


0 


0 


0 


0 


0 


0 


0 


0 


183 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


183 


0 


0 


0 


0 


Control 

Kef. 

KcswU: 










• 






















-1 


1 


n 


0 


34 


9 


1 


0 


0 


1 


0 


25n 


34 


<» 


43 




2 


0 


0 


■ 0 


0 


0 


0 


0 


0 


0 


0 


296 


0 


0 


0 


0 




0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


296 


0 


0 


0 


0 


ERIC 4 


0 


0 . 


0 


0 


0 


0 




0 


0 


0 


296 


0 


o' 


0 





AfiENCy_#3_Atlant1c City Vocational TABLE 3-12. REFERRAL, PLACEMENT AND FOLLOW-UP DATA FROM 



oc 



I 

CO 



Experimental 


Cluster' 
Number 


Date 
Hired 


Date 
r«rn)1 nated 


Grad Tr 
Yes 


aining 
No 


Date 
Grad. 


Ref. 1 


30 


6 


6 


0 


0 


0 


2 


3 


1 


1 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


4 


0 


-0 


0 


0 


0 


0 


Control 














Ref. 1 


- 

37 


1 


1 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


L 


0 


4 


0 


0 


0 


0 


0 


0 





Still 
Yes 


Job Fo1 
Working 
[ No 


lowup 
[late of 
FollowuD 


Still in 1 
Yes 


Training F 
raining 
f NO 


o1 lowup 
Training 
Yes 


Grad. 
No 


Date of 
Followup 


Training 
Grad. Date 


Experimental 
Followup 


8 


3 


11 


11 


15 


13 


13 


26 


6 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control 1 
Fol 1 owup 


0 


2 


2 


0 


0 


0 


0 


0 


0 


2 


0 


0 


0 






0 


0 


0 


0 


3 


0 


0 


0 


: 


0 


0 


0 


0 


0 



ERIC 



AGENCY I S Hew Bru nswick: Vocational TABLE 3-13. REFERRAL, PLACEMENT AND FOLLOW-UP DATA FROM 
Rehabilitation THE NEW JERSEY/CLEFF EXPERIMENT 





Experimental 


Control 


Tout Processed 


125 


126 


Total Referred 


67 {53.6) 


35 f27 8) 


Client Status: 
Tending Referral 


0 


0 


Pending Job Acceptance 


0 


0 


Pending Training Accept. 


0 


0 


Accept Intv * vb 


50 (40.0) 


31 (24.6) 


Accept Training 


10 ( 8.0) 


2 M.61 


Closed Not Referred 


58 (46.4) 


91 (72.2) 


Ifot Accepted by £inp1cryer 


0 


0 


Hot Accept by Train. 


0 


0 


Closed-Rfd-riot Accept. 


7 ( 5.6) 


2 (1.6) 


Out of 'Range 


0 


0 



Referral Result Codes 





Pend. 
Ref. 


Pend. 
•lob 

Accept > 


fend. 
Acceot. 


Acc< 
•lob 


ipted 
Train Inq 


By 

Gnol. 


Ko 

By 
Trttn. 


t Accepted 
By CI 
Veltd . 


tept 
NonV. 


Unknown 


Kissing 
& Out of 
tUnge 


Ref en 

Job 


'4l Code 
Trelnlnq 


Toul 

NOS. of 

Kef. 


Huiiber 
of Ref. 
Dates 


Exp. 
Ref. 

Result: 
































t 


0 


0 


0 


48 


12 


2 


0 


0 


1 


0 


62 


49 


13 


62 


61 




0 


0 


0 


1 


2 


0 


0 


0 


0 


0 


122 


1 


2 


3 


3 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


125 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


125 


0 


0 


0 


0 


Control 

Ref. 

Result: 
































■ t 


0 


0 


0 


25 


6 


0 


0 


0 


0 


0 


95 


26. 


5 


31 


32 


2 


0 


0 


- 0 


4 


0 


0 


0 


0 


0 


0 


122 


4 


0 


4 


4 


o i 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


126 


0 


0 


0 


0 


ERIC* 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


126 


0 


0 


0 


0 



A6EWC Y #5 N gw Brunswick Vocational TABLE 3-13. 



DATA ABOUT R£Fl 



;ab1 



litation 



REFERRAL, PLACEMENT AND FOLLOW-UP FROM THE 
NEW JERSEY/CLEFF EXPERIHEKT 



Experimental 


Cluster* 
Number 


Date 
Hired 


Date 
reml nated 


Grad Tr 
Yes 


aining 
No 


Date 
Grad. 


Ref. 1 


43 


0 


0 


0 


0 


0 


2 


2 


0 


0 


0 


0 


0 


3 


0 


0 


, 0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


Control 














Ref. 1 


19 


6 


1 


1 


0 


1 


2 


3 


1 


0 


1 


0 


1 


3 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 





Still' 
Yes 


Job Fol 
Working 
No 


I lowup 
1 Uate of 
Fol lowup 


Training F 
Still In Training 
Yes 1 No 


ollowup 
Training 
Yes 


Grad. 
1 No 


Date of 
Fol lowup 


Training 
Grad .Date 


Experimental . 
Fol lowup ' 


17 


2 


19 


4 


5 


3 


4 


9 


4 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control , 
Fol lowup 


19 


1 


20 


2 


3 


0 


3 


5 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 



ACEHCr I 6 PPrth Am boy Model TABLE 3-14. REFERRAL. PUCEMEtfT AND FOLLOW-UP DATA FROM THE 



Cities 



NEW JERSEY/CLEFF EXPERIMENT 



1 CD 

5J to 





ExDcr 


Inental 


Control 


Tat«1 Processed 


233 


225 


Total Referred 


158 


(67.8) 


108 


(48.0) 


CHent SUtus: 
PendtnS Referral 


17 


( 7.3) 


0 


Peftdtns Job Acceptance 


1 


f 0.4^ 


1 


f 0.4) 


Pending Tratning Accept. 


0 


0 


Accept Into Job 


104 


(44.6) 


68 


(30.2) 


Accept Training 


15 


( 6.4) 


5 


{ 2.2) 


Closed not Referred 


58 


(24.9) 


117 


(52.0) 


Mot Accepted by E.p1oyer 


0 


0 


Not Accept by Train. 


0 


0 


Closed-Rfd-IJot Accept. 


38 


(16.3) 


34 


(15.1) 


Out of 'Range 


0 


0 



Referral Result Codes 





Pend. 
Ref. 


Pend. 
Job 

Accept. 


Pend. 
Tmg. 
Ac^t. , 


AcC( 

Job 


ipte^ 
Trainings 




..0 

By 

Train. 


t Accepted 
By CI 
Valid . 


ient 
HonV. 


Unknown 


Hissing 
( Out of 
Ranoe 


(tefert 
Job 


■«1 Code 
Trtininq 


Total 
tfos* of 
Ref. 


of Ref. 
Uates 


Result: 
































1 


0 


1 


0 


109 


19 


10 


0 


4 


4 


6 


80 


134 


22 


156 


158 


Z 


0 


0 


0 


13 


3 


11 


0 


5 


1 


5 


195 


35 


3 


38 


38 


3 


0 


0 


0 


1 


0 


7 


0 


0 


2 


0 


223 


10 


0 


10 


10 


< 


0 


0 


0 


0 


0 


2 


0 


G 


0 


0 


231 


2 


0 


2 


2 . 


Control 

Ref. 

Result: 










• 






















^1 


0 


0 


3 


65 


8 


22 


1 


3 


1 


4 


118 


95 


12 


107 


in? 


Z 


0 


0 


- 0 


11 


0 


5 


0 


0 


0 


0 


209 


16 


0 


16 






0 


0 


0 


0 


0 , 


1 


0 


0 


0 


0 


224 


1 


0 


1 


1 




0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


225 


0 


0 


0 


0 



AGENC Y #6 Perf h Amboy Hodel 



TABLE 3-14. REFERRAL. PLACEMENT AND FOLLOW-UP DATA FROM THE 



DATA ABOUT R£F£ 



Experimental 


Cluster' 
Number 


Date 
Hired 


Date 
reminated 


Grad Tr 
Yes 


aining 
No 


Oate 
Grad. 


Ref. 1 


81 


0 


0 


0 


0 


0 


2 


19 


0 


0 


0 


0 


0 


3 


6 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


Control 














Ref. 1 


96 


38 


3 


1 


1 


5 


2 


13 


7 


0 


0 


0 


0 


3 


1 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 





Still 

Yes 


^ob fol 
UorkinS 
No 


Ilowup 
Date of 
Followup 


Still in 1 
Yes 


Training F 
'raining 
No 


DllOWUP 

Training 
Yes 


Grad. 
No 


Oate of 
Followup 


Training 
Grad. Date 


Experimental i 
Followuo 


74 


19 


93 


8 


7 


3 


11 


15 


4 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control 
Followup 


44 


8 


52 


2 


6 


1 


6 


8 


5 


2 


2 


1 


3 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 1 0 


0 


0 


0 


0 



AGEMCY # 7 TOPS TA8LE 3-15. REFERRAL, PLACEMENT AND FOLLOW-UP DATA FROM THE NEW JERSEY/ 

CLEFF EXPERIMENT 



CO 





ExD^rlMntsl 


Contra 1 


ToUl Processed 


233 


107 


lou 1 itcrerre\] 


172 


03 


client ^tafiK* 

Pending Referral 


n 


{ 3.9) 


0 


Pending Job AccepUftce 


0 


0 


Pending Training Accept. 


0 


1 


f 0.9) 


Accept Into Job 


34 


(12.0) 


9 


( 8.4) 


Accept Training 


54 


(19.1) 


15 


(14.0) 


Closed Not Referred 


100 


(35.3) 


44 


(41.1) 


Not Accepted by Einployer 


3 


( 1.1) 


0 


Not Accept by Train. 


30 


(10.6) 


0 


Closed-ftfd-Ifot Accept, 


51 


(18.0) 


38 


(35.5) 


Out of Range 


0 


0 



Referral Result Codes 





Pcnd. 
Ref, 


Pend. 
Job 

Accept. 


Pend. 
Tmg. 
A^cwt. . 


Acc« 
Job 


ipted 
Tr«}nlM 


ay 


No 

By 
Train. 


t Accepted 
By CI 
Valid . 


lent 
NonV. 


Unknown 


mssing 
t Out of 
Ranqe 


Refen 
Job 


■«1 Code 
Training 


TotaJ 
Nos. of 
Ref. 


Humber 
of Ref . 
OiteS 


Exp. 
Ref. 
Result: 
































1 


0 


0 


0 


38 


108 


0 


20 


0 


0 


0 


117 


38 


134 


172 


172 


2 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


280 


3 


0 


3 


3 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


283 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


283 


0 


0 


0 


0 


Control 

Ref, 

Result: 










• 
























0 


0 


1 


10 


10 


0 


22 


5 


0 


7 


52 


13 


40 


53 


51 


2 


0 


0 


' 0 


0 


0 


0 


0 


0 


0 


0 


107 


0 


0 


0 


0 




0 


0 


0 


0 


0 


0 


0 


0 


li 


0 


107 


0 


0 


0 


0 




0 


0. 


0 


0 


0 


0 


0 


0 


0 


0 


107 


0 


6 


0 


0 



AGENC Y #7 TOPS 
DATA ABOUT REFERRALS 



TABLE 3-15. REFERRAL. PUCEMENT AND FOLLOW-UP DATA FROM THE NEW JERSEY/CLEFF 
EXPERIMENT 



Experimental 


CI uster' 
Nundier 


Date 
Hired 


Date 
fenni nated 


Grad Tr 
Yes 


aining 
No 


Date 
Grad. 


Ref. 1 


142 


0 


0 


0 


0 


0 


2 


2 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


Control 














Ref. 1 


35 


0 


0 


0 


0 


0 


Z 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 





stni 

Yes 


Oob Fol 
Working 
1 No 


1 1 0MUp 

Date of 
Followup 


Still in 1 
Yes 


Training F 
'raining 
No 


ollowup 
Training 
Yes 


Grad. 
1 No 


Date of 
Followup 


Training 
Grad. Date 


Experimental 
rollowup 


34 


7 


41 


54 


69 


11 


112 


123 


23 


Z 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control , 
Followup 


1 


4 


5 


1 


0 


0 


0 


1 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 



AGENCY # 8 FQCJS TABLE 3-16. REFERRAL, PLACEMENT ANO FOLLOW-UP OATA FROM THE NEW JERSEY/CLEFF 

EXPERIMENT 





Experimental 


Control 


Total Processed 


95 


95 


locti Kcrerrea 


60 


(63.2) 


37 


(38.9) 


Client status* 
Pending Referral 


12 


(12,6) 


2 


( 2,1) 


Pendtng Job Acceptanct 


3 


( 3.2) 


0 


Pending Tralntng Accept. 


5 


( 5,3) 


0 


Accept Into Job 


28 


(29.5) 


15 


(15,8) 


Accept Training 


10 


(10.5) 


7 


(7.4) 


Closed Hot Referred 


23 


(24.2) 


56 


(58.9) 


Sot Accepted by Employer 


0 


0 


Hot Accept by Train. 


0 


0 


Closed-Rfd-rtot Accept. 


14 


(1^-7) 


15 


(15.8) 


Out of Range 


0 


D 



Referral Result Codes 





Pend. 
Ref. 


Pend, 
Job 

Accept. 


Pend. 
Tmg. 
Accepts 


Accc 
Job 


pted 

Train Inq 


By 


»o 

By 
Train. 


t Accepted 
By Cl 
Valid . 


lent 

NonV. j 


Unknown 


Hissing 
t Out of 
Range 


Ref err 

Job 


'al Code 
Training 


Total 
Nos. of 
Ref. 


Number 
of Rcf. 
uates 


Exp. 
Rtf. 
Result: 
































1 


0 


3 


5 


28 


8 


8 


1 


4 


1 


1 


36 


38 


15 


53 


60 


2 


0 


0 


0 


0 


1 


1 


0 


1 


1 


0 


91 


3 


1 


4 


4 


3 


0 


0 


0 


0 


0 


0 


0 


0 


■ 0 


0 


95 


0 


0 


0 


D 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


95 


0 


0 


0 


0 


Control 

Ref. 

Result: 










• 






















■ 1 


0 


0 


0 


14 


7 


11 


2 


2 


0 


1 


58 


27 


10 


37 


39 


2 


0 


0 


• 0 


1 


1 


1 


0 


1 


1 


0 


90 


3 


2 


5 


5 


O 3 


0 




fl 


0 


0 


P 


0 


0 


0 


0 


95 


0 


1 


1 


1 


ERIC 4 


0 


0. 


0 


0 


0 


0 


0 


0 


0 


0 


95 


0 


0 


0 


0 



A6ENCy_J8_£0CUS 
DATA ASOUT REFERRALS 



TABLE 3-16. REFERRAL, PLACEMENT AND FOLLOM-UP DATA FROM THE NEW JERSEY/CLEFF 





EXPERIME^ 


nr 


Experimental 


Number 


Date 
Hired 


Date 
Temtinated 


Grad Tr 
Yes 


aining 
Ho 


Date 
Grad. 


Ref. 1 


26 


0 


0 


0 


0 


0 


2 


1 


2 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


Control 














Ref. 1 


11 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 






Still 
Ye$ 


Job Fo1 
Working 
NO 


lowup 
{late of 
Fol 1 owup 


Still in 1 
Yes 


Training F 
'raining 
1 No 


oil owup 
Training 

Yes 1 


Grad. 
1 No 


Date of 
Followup 


Training 
Grad. Date 


Experimental 
Foi lowup * 


36 


0 


48 


11 


0 


0 


0 


15 


0 


2 


1 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Control , 
Followup 


0 


0 


0 


• 

0 


0 


0 


0 


0 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


3 


0 1 0 


0 


0 


0 


0 


0 


0 


0 
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B'lock^). This block contains one data item from block U of 
the data collection form; i.e.» the cluster number 
that best represents the job or training program 
to which the client was referred. 

The other entries In block(|)coffle from block V of 
the data collection form. These are the hire date* 
the tenninatlon date* and— If the client was placed 
in a training program—a code to indicate If the 
client graduated and* if so* the date of graduation. 
One should note that the entries in block V of the 
';!ata collection form were linked to the referrals 
recorded In block U by the referral line number. 

Block The entries in this block come from blocks W and X 
on page 2 of the data collection form. They are 
all concerned with follow-up data. While not ex- 
plicitly shown on the data collection form* the 
referral line number was entered prior to the 
follow-up data to indicate which referral the 
follow-up pertained to. 

One should note that the client's current status 
is supposed to reflect the latest status. In other 
words* over the course of the experiment* a client 
might have been referred to and accepted into a 
training program* graduated* and then referred to 
and accepted into a job. Thus the client's current 
status would be Accepted Into Job but there would 
be two referrals recorded for this client. 



3.2,2.1 Discussion of the Referral. Placenent and rollow-uD Data 

Taken at face value* the data In block0of Table 3-9 shows that approxi- 
mately the same percentage of experimental and control group clients 
were placed into jobs. Examining the individual agency data* one sees 
that there is considerable variation in these numbers for each agency. 

For the experimental group* the percent placed into a job varied between 
a low of 11.95S for BICC to a high of 44.6% for the Perth Amboy Model 
Cities agency. For the control group* the percent placed into a job 
varied between a low of 8,4% for TOPS to a high of 42,3% for the Atlantic 
City Model Cities agency. 
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Table 3-17 shows the total distribution of referral results for the 
experimental and control groups. The percentage distribution is 
also shown and is then adjusted to remove those referrals whose 
result is listed as pending referral, pending job acceptance or pend- 
ing training acceptance. Since the referral result data has not been 
disaggregated by the type of referral (to job or training) we are 
going to have to make some estimate of the total job and training 
referrals to obtain the percent of each which were successful As 
mentioned earlier, the data item that indicates the type of referral 
shows that of the 1355 experimental group referrals, recorded by type, 
63.8% of these were to jobs and 36.2% to training. For the control 
group, the data show that of the 429 referrals, recorded by type, 80.2% 
were to jobs and 19.8% to training. 

Applying these percentages to the adjusted totals shown in Figure 3-17, 
one obtains an estimate of the referrals to jobs or training that the 
referral results pertain to. These estimates (and the data used to 
generate them) are shown in Table 3-18 for each agency and for the total 
experimental and control groups. Thus the estimated job and training 
referrals against which the results will be applied are: 

1) Experimental group: 844 job and 429 training referrals 

2) Control group: 346 job and 85 training referrals 

These estimates combined with the number of successful referrals yield 
the placement/referral estimates shown in the last two columns of 
Table 3-18. 

Thus one sees that the percentage of successful job referrals is approxi 
mately the same for the two groups and that the percent of successful 
training referrals is higher for the experimental group. 

Examination of the individual agency data shown in Table 3-18 shows the 
fol lowing: 

1) For the three agencies, ACHC, ACVR, AND NBVR, the total number 
of referrals and the number of successful referrals are almost 
the same. For these three agencies, the total number of 
successful referrals (to jobs and training) is 186 for the 
experimental group and 90 for the control group. There were 
13 experimental group referrals which resulted in nonaccepted 
results and 2 such referrals from the control groups. 
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TABLE 3-17. SUMMARY OF REFERRAL RESULT DISTRIBUTION 



Experimental 
% of Adjusted 

Referral Result Total Total Percent 



Pending Referral 


1 
1 


• uu \ 


Y 
A 


Pending Job Acceptance 


16 


.012 


X 


Pending Training Acceptance 


9 


.007 


X 


Accepted Into Job 


505 


.374 


.382 


Accepted Into Training 


364 


.270 


.?75 


Not Accepted: 








• By Employer 


242 


.179 


.183 


• By Trainer 


66 


.049 


.050 


• By Client 








-Valid Reason 


22 


.016 


.017 


4tonvalid Reason 


65 


.048 


.049 


-Reason Unknown 


59 


.044 


.044 


Total 


1349 






Adjusted Total 


1323 







Control 

% of Adjusted 
Total Total Percent 



1 
I 


.lilic 


v 

A 


0 


0 


X 


4 


.009 


X 


215 


.493 


.499 


42 


.099 


.097 


112 


.257 


.260 


30 


.069 


.070 


11 


.025 


.026 


9 


.021 


.021 


12 


.028 


.028 
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431 
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TABLE 3-18. ESTIMATES OF PLACEMENT PER REFERRAL RATIOS 



EC 






[ 






















1 








Referrals By Type 


X Distribution 


Total Referrals 


Estimate Referrals 


Referral Results 


X Successful 








To 


To 




To 


To 


With Results 


To 


To 


Accepted Into 


To 


To 








Job 


Training 


Total 


Job 


Training 




Job 


Training 


ODD 


1 r4 1 n t ny 


Job 


Training 




BICC 


Expert mental 


i 


990 






. J 1 D 


69B 




999 


210 


142 


.•til 


.D*tU 




Control 


116 


5 


121 


.959 


.041 


122 


117 


5 


40 


0 


.342 


0.000 




ACMC 


Experimental 


17 


27 


44 


.386 


.614 


43 


17 


27 


16 


27 


.941 


1.00 




Control 


9 


1 


10 


.900 


.100 


12 


n 


1 


11 


1 


1.00 


1.00 




ACVR 


Experimental 


43 


45 


88 


.489 


.511 


90 


44 


46 


38 


42 


.864 


.913 




Lon L r o 1 


34 


9 


43 


.791 


.209 




36 


9 


34 


9 


.944 


1.0O 




NBVR 


Experimental 


50 


15 


65 


.76& 


.231 


66 


51 


15 


49 


14 


.961 


.933 




Control 




c 




.03/ 


• 1 H J 


35 


•»n 


c 


29 


6 




n 9n\ 




PAMC 


Experimental 


181 


25 


206 


.879 


.121 


202 


17B 


24 


123 


22 


,691 


.917 




Control 


112 


12 


124 


.903 


.097 


121 


109 


12 


76 


8 


.697 


.667 




TOPS 


Experimental 


41 


134 


175 


.234 


.766 


169 


40 


129 


41 


108 


(1.025) 


.837 




Control 


13 


40 


53 


.245 


.755 


54 


13 


41 


10 


10 


.769 


.244 




FOCUS 


Experimental 


41 


tc 


57 


.719 


.281 


55 


40 


15 


28 


9 


.700 


.600 




Control 


30 


13 


43 


.698 


.302 


42 


29 


13 


15 


8 


.517 


.615 




Total 


Experimental 


864 


491 


1355 


.638 


.362 


1323 


844 


479 


505 


364 


.59B 


.760 




Control 


344 


85 


429 


.802 


.198 


431 


346 


85 


215 


42 


.621 


.<}94 



These data are hard to accept at face value. Ultrasystems 
does not know if it is a correct reflection o^ what really 
happened or is due to a failure to record all the referrals 
made for each client. One should bear in mind that the 
data was not collected by Ultrasystems. In addition, it is 
Ultrasystems' opinion that these agencies, staffed In whole 
or in part by Vocational Rehabilitation personnel, do not 
rigorously record placement-related events In ttie same manner 
as, say, the U.S. Employment Service does. In other words, only 
successful referrals may be recorded even though others are made. 
One should also note that placement of Vocational Rehabilitation 
clients is usually done through a direct job development process 
and not from open order job listings. 

2) The percentage distributions of referrals by type Is quite 
different between agencies and between the experimental and 
control groups for each agency. Thus, for example, the BICC 
agency had approximately a 2-to-l split in job and training 
referrals. The control group for this agency, which was drawn 
from the Urban League in the Newark area, shows an almost total 
concentration on job referrals. BICC accounts for 46.3% of all 
experimental group training referrals and 39.016 of all accepted- 
in-training referral results. Its control group accounts for 
5.016 of all control group training referrals and O.OX of control 
group referrals resulting in acceptance Into training. 

The TOPS agency shows the highest percentage of referrals to 
training (for both the experimental and control groups) of all 
the agencies. Approximately 75% of all referrals from both 
groups were to training. In combination with BICC, these two 
agencies account for 73.3% of all experimental group clients 
referred to training , and 68.7% of all experimental group 
clients accepted into training. They account for 54.116 of 
all control group clients referred to training and 23.816 ac- 
cepted Into training. 

3) Ignoring the issue raised In point 1 above, one obtains* from 
the data shown in Table 3-18, the following referral and 
referral result distributions from the five agencies excluding 
BICC and TOPS: 





Estimated Referrals 


Accepted Into 


Percent of 
Referrals Accepted 




To Jobsj 


To Training 


Job 


Training 


Job 1 


Training 


Experimental 


328 


128 


254 


114 


.774 


.891 


Control 


216 


39 


165 


32 


.764 


.821 
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TABLE 3-19. CLIENT INDICES FOR EXPERIMENTAL GROUP CLIENTS; NEW JERSEY/aEFF EXPERIMENT 





Client Adjustment Index 


Client Difference Index 


Agency 
(Total Exp- 
el ients^ 


No. of 
Clients 


Mean 


Std.Dev. 


Min. 


Max. 


No. of 
Clients 


Mean 


Std.Dev. 


Min. 


Max. 


BICC 
(1561) 


1278 


50.674 


34.003 


-74.0 


+100.0 


1318 


901.747 


586.674 


1 


4775 


ACMC 
(139) 


131 


35.518 


38.678 


-61,0 


+ 96.0 


135 


910.585 


598.529 


64 


2969 


ACVR 
(183) 


178 


40.758 


34.612 


-45.0 


+100.0 


180 


879.622 


536.582 


109 


2651 


NBVR 
(125) 


118 


36.246 


37 . 322 


-57.0 


+ 99.0 


119 


855.555 


520.827 


7 


2530 


PAMC 
(233) 


219 


35.516 


42.049 


-75.0 


+100.0 


221 


817.593 


900.864 


10 


9333 


TOPS 
(283) 


265 


36.585 


34.344 


-67.0 


+ 97.0 


272 


692.926 


427.992 


29 


3268 


FOCUS 
(95) 


86 


44.826 


36.739 


-55.0 


+ 97.0 


95 


558.158 


422.797 


65 


2034 


TOTAL 
(2619) 


2275 










2340 











Earlier it was mentioned that no CJMS score data was analyzed with 
regard to the clients and the referrals made on their behalf. Actu- 
ally, Ultrasystenis did obtain from the data received the distribu- 
tion of the client's adjustment and difference indices, Ttiis data 
is presented in Table 3-19, The table shows all the data received. 
The data has not been disaggregated into cells based on whether 
the client was or was not referred to a job or a training program 
(or both) and if referred was or was not placed. Thus it is not 
possible to determine from the data shown in the table to what 
degree the client adjustment index criteria were adhered to. One 
sees from the data that for each agency the number of clients for 
which the two indices were provided is not the same nor is it equal 
to the total experimental group clients. The difference between 
the number of clients for which the adjustment inoex and the differ- 
ence index are known is due to non-reporting and is troublesome. 
The difference between the total experimental clients processed 
and the total for which these indices were reported could be and 
probably is due to the fact that some clients only take the prefer- 
ence SICL and hence these indices are not obtainable. The experi- 
mental criteria on minimum scores for referral consideration did not 
explicitly indicate what is to be done for these clients. Appar- 
entlyi the match index criteria would be the only criteria utilized. 

In summary* the data obtained and the analyses completed do not 
provide adequate information from which any conclusions regarding 
the effectiveness of the use of the CJMS in the referral process 
can be drawn. The problems with the data, as collectedi are, in 
Ultrasy stems' opinion, due to 

1) a low credibility due to the lack of a definitive control 
group sampling plan and the use of clients from a different 
agency for one of the control groups 

2) in lieu of the above* a failure to run the experiment in 
the time-phased manner as originally designed 

3) failure to obtain JOCLs on each job or training program 
associated with the referrals and placements made. In 
the absence of unique JOCLs the experimental data base 
failed to capture enough job or training program descrip- 
tions so that a cluster profile could be reasonably well 
assigned. 

4) an apparent lack of complete referral data on each client 
leading to a lack of credibility In the placement/referral 
ratios 
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5) an experimental design that tended to Ignore the substan- 
tial differences in the manner in which the participating 
agencies provide services to people* especially as regards 
job and/or training referrals 

6) an experimental design that placed too much emphasis on 
the completion of the SICLs as the sole causal link in the 
comparison of placement performance. This could have been 
alleviated to some degree if the control group clients 
also took the SICLs and their resultant match and client 
scores were available for analysis. This would be done in 
such a manner that the scores for this group would not be 
made known to the agency staff. This suggestion ignores 

the ethical problem involved with havina a person spend about 
two hours of their time completing the SICLs solely for the 
purpose of research. !f such an approach had been utilized* 
even for a sample of the total control group* it would en- 
able one to compare on a more direct basis the referrals 
that are selected* from the alternatives available* using 
either the "traditional" agency staff approach or the 
approach assisted by the CJMS. 



3.3 REVIEW OF THE FINAL STATUS REPORT ON 

THE NEW JERSEY/CLEFF EXPERIMENT SUBMITTED 
BY ADP-POS, INC. 

The New Jersey/Cleff experiment was conducted by Personnel Data 
Systems* a subsidiary of ADP Inc.* under contract to the Office of 
Economric Opportunity. The contract called for status reports to 
be submitted every three months during the course of the experiment. 
The final status report was submitted in June of 1973.1 

The periodic status reports presented data on the experiment's 
progress in terms of total clients processed {in the experimental 
and control groups) and bre&ltouts of tljeir then current status 
using the definitions shown in block Q of Table 3-9. In addition, 
these reports discussed aspects of the various participating agency's 
involvement in the experiment. The statistical activity data 



The final status report was submitted to the Office of Economic 
Opportunity as Deliverable Item 1B2-6 under OEO Contract BIC-5261 
The report has not been included as an appendix because of its 
size. 
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EXHI8IT 3-1 














TABLE 1-4 


1 


-10 








SUMHARY OF CJHS PERFORMANCE 


oT AbtnifT 










CONTROL (CTRL) VS. EXPERIMENTAL 


(EXP) GROUP 










Focu; 


3 (8) 


8ICC (1) 








CTRL 


EXP 


aRL 




EXP 




1 ) Total Processed 


95 (95) 100% 


95 (95) 100% 


295 (295) 100% 


1,560 


(1561) 100% 


cj Total Referred 


37 (37) 39% 


60 (60) 63% 


113 (113) 38% 


599 


(599) 


38% 


3) Peiidiiig Ref. 


2 (2) 2% 


12 (12) 13% 


(0) 


147 


(147) 


10% 


4j Closed} Hot Reft 


56 (56) 59% 


23 (23) 24% 


182 (182) 62% 


814 


.(814) 


52% 


5) Accepted 


22 (22) 23% 


38 (38) 40% 


39 (39) 13% 


258 


(257) 


17% 


6) Not Accepted 


15 (15) 16% 


14 (14) 15% 


74 (74) 25% 


330 


(331) 


21% 




PERTH AH80Y (6) 


TOPS (7). 








CTRL 


EXP 


aRL 




EXP 




1 / 1 Ota 1 rrocesseo 


224 (225)100% 


233 (233)100% 


107 (107)100% 


283 (283) 100% 


lotai KeTerrea 


107 (108) 48% 


158 (158) 68% 


63 (63) 59% 


173 (172) 


61% 


0} renaing hbt* 


(0) 


17 (17) 7% 


(0) 


9 


(11) 


3% 


*t) LioseO} nor KeT» 


117 (117) 52% 


58 (58) 25% 


44 (44) 41% 


101 (100) 


36% 


5) Accepted 


73 (73) 33% 


119 (119) 51% 


24 (24) 22% 


89 


(88) 


31% 


6) Not Accepted 


33 (34) 15% 


38 (38) 16% 


38 (38) 36% 


84 


(84) 


30% 




ATLANTIC CITY {2Hf3^ 


NEW BRUNSWICK (5) 








CTRL 


EXP 


CTRL 




EXP 




I) loiai rrocessea 


322 (322)100X 


322 (322)100% 


126 (126)100% 


125 (125) 100% 


c } \ Ota 1 KeTerrea 


59 (55) 18% 


129 (130) 40% 


35 (35) 28% 


67 


(67) 


54% 


0 } r ena i ng kct . 


(0) 


52 (50) 16% 


(0) 




(0) 




H ^ 1 uscU } not VXT • 


263 (267) 82% 


141 (141) 44% 


91 (91) 72% 


58 


(58) 


46% 


5) Accepted 


55 (51) 17% 


107 (108) 33% 


33 (33) 26% 


60 


(60) 


48% 


6) Not Accepted 


4 (4) 1% 


22 (22) 7% 


1 2 (2) 2% 


7 


(7) 


6% 
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EXHIBIT 3-2 



1-12 

TABLE 1-5 



SUMMARY OF CJMS PERFORMANCE BY AGENCY AS A 
FUNCTION OF TOTAL REFERRED 







FOCUS 






BICC 










CTRL 


EXP 






CTRL 




EXP 




Total Referred 


37 


100% 


60 


100% 


113 


100% 


599 




100% 


Accepted 


22 


59% 


38 


64% 


39 


35% 


258 




43% 


Pending Accept. 




(0) 


8 (8) 


13% 




(0) 


n 


(11) 


2% 


Not Accepted 


15 


41% 


14 


23% 


74 


65% 


330 




55% 






PERTH AMBOY 






TOPS 










CTRL 


EXP 






CTRL 




EXP 




Total Referred 


107 


100% 


158 


100% 


63 




173 




100% 


Accepted 


73 


68% 


119 


75% 


24 


38% 


89 




51% 


Pending Accept. 


1 (1) 1% 


1 (1) 


1% 


1 (1) 2% 




(0) 




Not Accepted 


33 


31% 


38 


24% 


38 


60% 


84 




49% 






ATLANTIC CITY 






NEW Bf 


tUNSWICK 








CTRL 


EXP 






CTRL 




EXP 




total Referred 


59 


100% 


129 


100% 


35 


100% 


67 




100% 


Accepted 


55 


93% 


107 


83% 


33 


94% 


60 




90% 


Pending Accept. 




(0) 


(0) 






(0) 




(0) 




Not Accepted 


4 


7% 


22 


17% 


2 


6% 


7 




10% 
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EXHIBIT 3-3 



TABLE 4.1 



SUW1ARY DATA FOR CONTROL AND 
EXPERIMENTAL GROUPS 



CONTROL EXPERIMENTAL 

Total Processed ^^^OS (1170) 1005S 2.636 (2619) 1005S 

Total Referred 439 (411) 365S 1,189 (1186) 45% 

Total Pending Referral 2 (2) 0% 237 (237) 9% 

Total Closed, Not Referred 767 (757) 64% 1,210 (1194) 46% 



TABLE 4.2 



DISTRIBUTION OF THOSE REFERRED FOR 
CONTROL AND EXPERIMENTAL GROUPS 



CONTROL EXPERIMENTAL 



Total Referred 439 (411) 100% 1,189 (1186) 100% 

A) Jobs 360 82% 761 64% 

B) Training 79 18% 428 36% 
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The data shown in Table 3-18 yields the following: 



Job 
Referrals 



Training 
Referrals 



Total 
Referral s 



Ratio: Total 
Referrals to 
Total Clients 
Referred 



Experimental 



844 (63. b; 



479 (36.2) 



1323 



1.12 



Control 



346 (80.3) 



85 (19.7) 



431 



1.05 



It is interesting to note that the percentage distribution of refer- 
ral transactions by type is approximately the same as the percent of 
clients whose current status code represents a referral to a job or 
a training program as shown in Exhibit 3-3. Probably this is due 
to the low ratio of referral transactions to individuals referred* 
i.e.) most clients who were referred at least once to a job or 
training opening were only referred this one time. 

Therefore, even though there are certain unresolved differences 
between the data presented in the MIP-POS Inc. final report and the 
data presented by Ultrasystems in this, report* the differences are 
not very substantial . 

ADP-POS Inc. readied the following conclusions or findings as shown 
in Exhibits 3-4 and 3-5. The gist of these statements is that 
through the use of the CJMS clients have a greater chance of being 
referred and that when referred have a greater probability of being 
accepted in either a job or a training program. The data we pre- 
sented in Tables 3-9 through 3-16 and the data presented in Exhibits 
3-1 through 3-3 do show that a larger percent of control group clients 
were in closed-not referred status (64.75! of the control group as 
compared to 45.6% of the experimental as shown in Table 3-9) and 
that a larger percent of the experimental group clients were accepted 
into jobs and/or training as compared to the control group (25. 6% 
compared to 20.7%). One should note that there were a considerable 
number of experimental group clients still in pending status. If 
one removed all pending referral clients from both groups* one Would 
find that the percent of clients in closed-not referred status would 
be 50.1% for the experimental group and 64.8% for the control* and 
the accepted into job^ or training would be 28.1% for experimental 
and 20.7% for control. 
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Exhibit 3-4 - 
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Table 1-4 SUMMARY OF CJMS PERFORMANCE BY AGENCY^ 
CONTROL (CTRL) vs. EXPERIMENTAL (EXP) GROUPS shows the 
following general characteristics: 

a) In all cases the Total Referred as a percentage 
of Total Processed (item 2) was higher for the 
Experimental than for the Control Groups, Even 
for BICC it is expected that a significant por- 
tion of those Pending Referral (item 3) will 
transfer to the Referral category. The range of 
absolute increase in percent Referred (except 
for BICC as noted above) is from 2% to 26%s 

the range of relative increase in percent re- 
ferred (Exp. % + Control %) is from 3% to J22%. 

b) A corrollary to a) above is the universal re- 
duction in the percentage of clients closed^ not 
referred (item 4)^ The most frustrating and per- 
haps damaging effect of many programs for the 
disadvantaged is that the individual gets no help 
at all and is again rejected. Through the use of 
the CJMS the group in this category was lower in 
every case for the Experimental Groups than the 
Control Groups, The range of absolute decrease 
in Closed not Referred is 20% to 38%. The range 
of relative decrease C( 1- Exp% + Control %} x 100} 
is 12% to 59%. 

c) Since the primary effort in this project was to 
place individuals into gobs or training programs 
leading to sobs the Accepted category (item $) 
becomes most important. In every instance the 
Experimental Groups^ using the CJMSj showed an 
increase in the Accepted category over the Control 
Groups. The range of absolute increase in the 
percent Accepted category is from 4% to 22%. The 
range of relative increase is from 31% to 94%. 
This is accomplished without significatly affecting 
the category of those referred but not accepted 
(item 6). 

^^In summary^ the use of the CJMS in this experiment has re- 
sulted in increased percentages of people referred and ac- 
cepted by employers without a significant negative or positive 
effect on those who were referred but not accepted. It has 
also produced smaller percentages of people closed^ not re- 
ferreds that is fewer people weve not helped at all* 

Note: Table 1-4 is shown as Exhibit 3-1. 
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Exhibit 3-5 



Table J-5 SUMMARY OF CJMS PERFORMAltCB AGBltCX AS A FUCTIOH 
OF TOTAL REFERRED is presented here to be consistent with 
prior PDS progress reports* The data, however, is more 
significant when evaluated as in Table As a prime ea?- 

ample of this, reference is made to the Atlantic City data 
in Table The percent of the Control Group Accepted is 

93% while for the Experimental Group Accepted it is 8S%; an 
apparent absolute decrease of 10%. However, when it is 
recognized that this is really 9S% of 18% Referred for the 
Control Group V3. 83% of 40% Referred for the Experimental 
Group the true difference is clear. The Experimental Group 
is outperforming the Control Group by almost two to one. 



Note; Table 1-5 is shown as Exhibit 3-2. 



''The importance of the summary data presented in Table 
is that there i^ <i significantly larger percentage, 18%, 
of applicants in the Control Group who were closed without 
having been referred either to jobs or to training leading 
to yobs. In all previous reports, the relatively high 
percentage of individuals "pending referral" did not allow 
drawing a difinite conclusion from this portion of the 
data. In this report, the "pending referral" percentage 
for the Control Group is 0% which means that this cate- 
gory can no longer influence, in a negative way, the 
"total referred" category. This positive trend, reflected 
in all previous reports and now definitely indicated, sig^ 
nifies that those applicants processed by the CJMS are 
more likely to be referred to either a job or training 
leading to a job as opposed to those processed without 
benefit of the CJMS. " 



Note; Table 4-1 is shown as Exhibit 3-3. 
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The major Issues, In Ultra systems' opinion* In Interpreting these 
numbers are: 

1) How were the control group clients selected? 

2) What are the characteristics of these two groups In 
relationship to both the agency services where they 
applied and the job or training openings available 
to these agencies? 

3) What were the operational procedures associated with 
the selection of clients to be given SICLs? 

4) How did the agencies use the CJKS? 

5) What were the CJMS scores associated with the experimental 
group clients and the referrals made on their behalf? 

It is not possible from the information that Ultrasystems obtained 
about the New Jersey experiment to definitively answer these ques- 
tions- It is very important^ however* that one understand the 
impact that these Issues have on the possible Interpretation of 
the data. Ultrasystems believes it knows enough about these issues 
so that there is serious doubt as to the validity of the data and 
especially as to its interpretation. The following, discussion will 
t^/iefly review what we learned regarding these Issues. 



Issue 1: Selection of Control Group Clients 

The original design for the selection of the control 
groups based on time phasing the use of the CJMS has 
already been discussed. The actual Implementation of 
the experiment did not adhere to this design and t^Ai 
control group clients were obtained by sampling the 
agency files. The exact manner in which this sampling 
was done Is not definitively known. We do know, 
however* that the entire control group to be used for 
the 8ICC agency was drawn from another agency. This 
group comprises 25.2% of the total control group and» 
has been shown, 1s» statistically* significantly differ- 
ent In demographic characteristics fran the 8ICC experi- 
mental group. Since the Intent of the control (or 
comparison) group is essentially to provide a means for 
assessing the relative effectiveness of the use of the 
CJMS, it is important that for all practical purposes 
the two groups differ only in the services provided 
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because of the use of the CJMS. Drawing a cbntrol 
group from another agency introduces the issue of 
agency differences independent of the use or ndtf-use 
of the WMS. 

The entire control group for TOPS was drawn from this 
agency's first training program. This training program 
had begun prior to the beginning of the experiment. 
Once the experiment began, all referrals to TOPS were 
given SICLs. The TOPS clients actually consist of two 
distinct groups, i.e., those who are enrolled in TOPS 
because they were referred to the training program by 
the local Model Cities or CEP programs, and those who 
more or less applied on their own accord. The Model 
Cities and CEP referrals had to be accepted unless TOPS 
could convince the CEP to withdraw the referral, an 
event which the TOPS staff could not remember happening. 
This mixture applies to both the experimental and control 
groups. The data received showed that 20.5% of the 
experimental group and 30.8% of the control group con- 
sisted of referrals from these agencies.^ The experi- 
mental group consisted of 190 individuals who came in 
on their own or were referred by friends (67.1%), whereas 
for the control group there were 49 such cases (45.8%). 
Thus, not only is there a distinct difference in the use 
of these two groups for con^arative purposes because of 
the issues of the time in which they were referred and 
enrolled (at the start of operations and after one year 
of experience and exposure) but in their composition as 
regards the manner in which they arrived at TOPS and were 
then screened for training. 

The manner in which the control group clients from the 
other agencies were selected is not known. The sampling 
methodology used at the Vocational Rehabilitation (VR) 
agencies is a serious issue since the clients accepted 
as cases by these agencies can have a wide range of 
physical or mental handicaps. The data shown in Tables 
3-4, 3-5, and 3-6 indicates that there is reason to be- 
lieve that there were significant differences in the 



These percentages are based on the data unadjusted for the cases 
with missing values. For the experimental group, there were 28 
cases with missing values (9.9%) and for the control group there 
were 20 cases or 18.7% of the total. 
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physical handicaps of the experimental and control 
groups frm these agencies. In addition, the VR 
agencies operate on a caseload system. In AOP-POS's 
first progress report, it was stated the control 
group sampling would be done on a rand(Kn basis by simply 
pulling from each agency's counselor files in order to 
insure that there was a general cross section of coun- 
selor activities represented. The progress report 
stated that "thus no individual counselor could bias 
the results obtained." No mention was made as to how 
this would relate to the eventual counselor distribution 
of the experimental group and no mention was made as to 
the issue of physical and mental handicaps and counselor 
specialization in such categories. The examination of 
the data received shows that the counselor distribution 
of experimental and control group caseloads per agency 
was not uniform and that in certain agencies one, two, 
or three counselors account for a large percentage of 
one of the groups and not the other* For example, in 
the Atlantic City Vocational Rehabilitation agencies 
(Numbers 2 and 3 on Table 3-1), there were three coun- 
selors who accounted fOr 35.4% of the experimental group 
and 22.5% of the control group. There were three other 
counselors who accounted for 35.7% of the control group 
and 18.9% of the experimental group. In the New Brunswick 
data base, there is one counselor who accounted for 27.2% 
of the experimental group and 0.0S5 of the control group. 
There were four other counselors who accounted for 52. OX 
of the experimental group and 38.8% of the control group. 
In Perth Amboy, the results were more balanced with three 
counselors accounting for 70.3% of the experimental group 
and 75.6% of the control. Even here, however, there was 
one counselor (of the three) who accounted for 24.g!t of-^.., 
the experimental group and 38.2% of the control group. 

Another facet of the control group selection has to do 
with the services provided by these agencies. This will 
be discussed in the next issue, but it should be men- 
tioned here that people who are currently employed are 
served by the Vocational Rehabilitation agencies with 
regard to their handicaps. While it is not definite that 
these agencies might not try to obtain a different job for 
these individuals, they represent a different type of client 
than the unemployed. The data received indicates that 100 
of the 673 control group clients frwn the four VR-associated 
agencies were employed at the time they were accepted as 
cases by these agencies. Only two of the 680 experimental 
group clients were employed at the time of acceptance. 
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overcome the obstacles to employment. In order to make 
such a otidgment» one really needs to know that there 
wasn't a significant difference in the level of obstacles 
to be overcome. Since the data does not enable one to 
ascertain this» and since there is at least some evidence 
that it wasn't true* the inferences cannot stand as is. 

There is one other aspect that is troublesome. Ultra- 
systems was told by the Vocational Rehabilitation agencies 
that cases were not closed by the agency until the client 
was successfully rehabilitated. Exactly what this means 
is not clear. We do know that if a client "disappears" 
or "refii&es to continue" the case will be closed. We also 
know that cases are considered successfully completed with- 
out the client obtaining a job either because the client 
already had a job or because obtaining a job was never 
in the plan developed. How this relates to the large per- 
centage who were closed-not referred and who were referred 
but not accepted is not clear. The inference made by 
ADP-PDS indicates that this is it» that the agency did 
not successfully complete its rehabilitation plan. This 
appears to Itltrasystems to be a contradiction with what 
we were told and that closing a case with no referrals 
would be a reflection of the client's "disappearance" or 
that the rehabilitation plan did not envision such actions. 

Issue 3: Selection of Clients for the Experimental Group 

This issue is similar to the first one discussed. The 
experimental design called for a time-phased usage of 
the CJMS based on agency quotas and on the administration 
of the SICLs to all clients applying at the agency who 
meet both the agency criteria and the CJMS criteria for 
placement. The control group cli''"*ts were to be those who 
met tbo agency criteria when, the Csf^lS^was not being used. 
The meaning of agency and CJMS criteria for placement 
had» to Itltrasystems' knowledge* never been defined. Each 
of the agencies had s<m& criteria. These included: 

1 ) screening out clients who did not appear to be rea- 
sonably serious or motivated 

2) rules governing the handicaps and income of clients 
who were eligible for services 

3) contractual arrangements whereby clients referred by 
certain agencies would be accepted. 
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The CJKS had certain criteria set forth. For one, 
the client had to possess a certain reading level 
to take the SICL's without resorting to oral adminis- 
tration. Dnce the clients were administered the SICL's, 
the resultant scores were to govern the referral process. 
To be referred to a job, the client was to have a minimum 
adjustment index of +.25 and a minimum match index of +.50. 

As has been discussed, the time-phased quota approach was 
not followed. Two of the agencies, BICC and TOPS, adminis- 
tered SICL's to all clients over the period of the experiment. 
One other agency, FOCUS, left it up to the client to decide. 
The maimer in which the other agencies selected clients is 
not very clear. The fact that jt is npt clear is a serious 
issue. Section 2 of this volume of the final report discussed 
this issue but did not contain any definitive statements on 
how the selection was actually done. The possibility that 
counselors were screening applicants for the administration 
of SICL's, based on some criteria, is a serious one in inter- 
preting the eventual outcomes. The fact that since oral 
administration of the SICL's is a very time-consuming process 
(and would not be and was not done in tnost instances because 
of the time involved) implies that care must be taken in 
selecting a control group because the experimental group 
will exclude people with low reading ability. 

Issue 4: How did the agencies use the CJhtS? 

This is actually the major issueMn light of the causal 
inferences drawn by ADP-POS from the data. This may sound 
awfully simplistic but to infer that the differences in 
the measures are due to the client taking the SICL's Is 
obviously unsound. The key to the causality is in the use 
of the system's scores in assisting the counselor in serving 
the client. The real key issue is measuring the difference 
between the services (actions) and their outcomes that were 
made on behalf of a client with and without the use of the 
CJHS. The experiment could have been at least partially run 
without ever making the CJMS results known and then the 
resultant services could have been analyzed vis-a-vis the 
CJMS scores. This, of course, eliminates the effect that 
the CJMS would have on the services provided but does provide 
a measure of the relationship between the services that a 
counselor would provide without the system and what the system 
would indicate. In any case, to infer causality implies that 
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the system was being used to at least assist in the 
referral to job or training process. There is, in 
Ultrasystems' opinion, some question as to what extent 
this really occurred. For instance, several of the 
agencies utilized aptitude, interest and psychological 
tests in addition to the Cleff system. The Perth Amboy 
agency is, in fact, a Diagnostic and Employability Center 
and utilized, in addition to the CJHS, the Singer/Graflex 
vocational (simulated work setting) evaluation system and 
written tests such as the Able, the California I and II, 
Of course, the use of tests developed to assess client 
aptitudes and interests is part of what the experiment 
referred to as "the traditional placement method," The 
use of multiple assessment techniques by agencies seeking 
to serve people with serious physical and mental handicaps 
is reasonably standard. How then was the CJMS used; i.e., 
what part did the CJMS play in the rehabilitation plan 
developed for these clients and the services provided? 
Were, in fact, all the clients in the two groups from each 
agency given all the same tests? Our understanding is that 
this was not so. Therefore, in addition to the use of the 
CJMS, there might well be differences between the two groups 
in the assessment techniques utilized. 

The TOPS agency used the CJMS in addition to the Metropolitan 
Achievement Test. Clients were screened, based on the 
results of the CJHS, the test and the counselor's interview. 
Some clients were accepted into training based on the results 
of this screening; others were accepted no matter what the 
results. The CJMS was used to assess clients' experiences 
and preferences in clerical-related dimensions. Clients were 
riot referred by TOPS to a training program; i.e., the clients 
referred and accepted by TOPS are in reality clients accepted 
by TOPS. Thus this is not really a measure of referral to 
training success. Closed-Not Referred includes those people 
that were not accepted by TOPS for training. In fact, the 
data presented by ADP^PDS and by us is misleading as to the 
TOPS control group. A report prepared by TOPS regarding 
their first training classes {and attached to ADP's second 
progress report) indicates tliat, in fact, 68 trainees were 
enrolled. Eleven of these dropped out during the semester, 
and twelve terminated during the job placement phase. The 
reasons for these terminations were given by TOPS, These 
included moving from the area, child care problems, health. 
Including pregnancy, death and illness in the family and 
personal difficulties. TOPS listed four people as terminating 
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because after exposure to clerical work on the job through 
placement they decided against it. In addition, not all 
those placed by TOPS from this group were placed into 
'jobs related to their training; i.e.* into clerical-related 
jobs. What the corresponding numbers are for the experi- 
mental group is not known. 

Thus the reasons for terminations go beyond the factors 
considered by the CJMS and the specific mention of the 
four students who felt clerical work was not for them 
indicates that prior to the use of the CJHS the screening 
being done was "not that bad." The turning around of the 
Interpretation of referred to training and accepted for 
TOPS is misleading. TOPS was not referring anyone to 
training. They were screening people for training. The 
fact that there were 68 people enrolled in the control 
group training is distinctly different than the data we 
received which indicates that 15 accepted training (based 
on the status code) and that 10 out of 40 training referrals 
accepted s^. Therefore* the CJMS was being used in screen- 
ing clients for acceptance and that the resul<:ant success of 
the services provided TOPS were considerably influenced by a 
myriad of other factors. Judging the success of the use of 
the CJMS for TOPS, based on those accepted into training, 
is misleading. The training program runs for several months 
and in that time considerable other information about each 
trainee is obtained and Personal relationships are established. 
All this is used in the eventual placement process. It is 
our understanding that the Cleff system was not used in the 
placement process. No jobs were profiled by TOPS. The even- 
tual placement of their trainees was done in essentially the 
standard manner. It is a distinct possibility that higher 
placement rates for the experimental group clients could have 
been achieved, not because they were better suited for clerical 
training as measured by the CJMS, but because the staff had 
gained valuable experience in screening on other factors, 
assisting with the client's personal situations and establish- 
ing better ties to employers. In addition, another report 
prepared by TOPS indicated that 32 OJT slots had been made 
available to them for their second semester clients. 

The discussion given in Section 2 of this volume of the final 
report discussed the issue of score interpretation being done 
by ADP-POS for FOCUS. It is Ultrasystems' opinion that the 
staff of this agency never really understood the CJHS and 
the interpretation of its scores. 
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It should also be mentioned that TOPS was not the only , 
agency which provided training to its clients. BICC, FOCUS 
and Perth Mioy also conducted their own training programs. 
Thus, again, the outccmie measures related to referred and 
accepted into training can be misleading. 

Issue 5: CJMS Scores 

The ADP-PDS final status report never mentions the issue of 
the scores associated with the clients and their job or 
training referrals. It assumes, apparently, that the basic 
criteria pertaining to the minimum levels of client adjust- 
ment and match indices were followed. Ultrasystems does not 
believe this assumption is valid. In addition, the question 
of how the CJMS job or training match score was determined 
when there was only one agency that used JOCL's is never 
addressed. Apparently, the inference is that the cluster 
scores were iised. Kow these cluster scores were translated 
into actual jobs or training matches is not clear. It is 
Ultrasystems' opinion, based on our interviews, that except 
for BICC this was not really done. The CJMS client scores 
were utilized to assist in assessing the client's preferences 
and experiences and this assessment was then utilized along 
with many other factors in the job or training referral process 



3.3.1 Summary of the Review of the ADP-PDS Final Status Report 

The above discussion all leads to Ultrasystems' basic conclusion that the 
interpretations drawn by M)P-POS} Inc, , based on the data presented about 
the comparative effectiveness of the CJMS and traditional agency place- 
ment methods cannot be justified by the data. It is our opinion 
that the data is not only incomplete and in many instances probably 
incorrect, but that the two client groups cannot be compared. It 
is also our opinion, therefore, that whatever the true measures of 
referral success were, the differences would be attributable to the 
differences in (1) the clients themselves; (2) their relationship 
to the labor market; and (3) the services they were provided in- 
dependent of the use of the CJMS. 

It is no easy task to execute a comparative experiment of this sort. 
However, it is Ultrasyst«ns' opinion that far too many issues were not 
dealt with or were ignored in the execution of this experiment. It is 
also our opinion that structuring an experiment to assess the canparative 
placement performance of the Cleff system compared to "traditional place- 
ment methods" is missing the point of the intent behind the CJMS. The 
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4.0 ORGANIZATION AND OPERATION OF THE SER/CLEFF EXPERIMENT 

The SER/Cleff experiment was substantially different in its objec- 
tives than the New Jersey experiment. The SER experiment was under- 
taken with the basic objective of obtaining data to analyze the re- 
lationship between the Cleff client-job match scores and the client's 
eventual retention. 

The basic characteristics of the SER/Cleff experiment were: 

1) The experiment was aimed solely at obtaining the data 
needed to analyze the relationship between the CJMS 
scores and the subsequent retention of the person on 
the job. 

2) The experiment was conducted by Ultrasystans in conjunc- 
tion with four SER project offices in California, located 
in Los Angeles, San Diego, San Jose> and Santa Ana. 

3) The experiment was not aimed at providing data to be 
used in a comparative analysis of referral, placement, 
and post-plaC3nent performance between a CJMS client group 
and a non-CJMS client group. 

4) The experiment did not require that the CJMS be used 

by the SER agency staff in assessing clients for enroll- 
ment in training and in the referral to job process. 

5) In conjunction with (4) above, no minimum client adjust- 
ment or job-match indice scores were established. 

6) The experiment obtained job profiles, using the JOCL, 
after the client who had taken the SICLs was placed. 

No effort was made to obtain JOCLs for use in the refer- 
ral process, i.e., JOCLs of open job orders. 

7) The experiment was conducted with the objective of 
administering SICLs to all SER clients applying at the 
agency or presently enrolled In training over a 4-month 
time period (March through June 1973). Clients would 
then be included in the foUowup data base as they were 
hired until a total of 150 clients was achieved. 
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Retention followup would be done solely through employer 
contact. Clients were not to be contacted. 

Retention followup was done at four time points* i.e., 
December 31, 1973, February 28, 1974* May 31, 1974, and 
December 10, 1974. 

The administration of the SICLs and the processing of 
them via the CJMS was the responsibility of Ultrasystems. 
In implementing this aspect of the experiment, a person 
was hired by each of the four SER offices whose only 
responsibilities were associated with the experiment. 
This person was employed by Ultrasystems. 

The obtaining of the job profiles was Ultrasystems' 
responsibility. 

The CJMS processing aspects of the experiment were set 
up as if the system was actually being used. The client 
profiles and cluster matches were available at each 
office on the day following the administration of the 
SICL. Communication of input and output data between 
the SER offices and Ultrasystems' computer facility 
was done via facsimile equipment. 

The remainder of this section will discuss some of the basic char- 
acteristics in more detail. Every point listed above will not be 
exhaustively discussed. The discussion to follow will concentrate 
on what were, in Ultrasystems' opinion, the more important aspects 
of the experiment. The activity and retention data associated with 
the experiment will be discussed in Section 5. 



4.1 BRIEF DESCRIPTION OF THE SER OFFICES AND 

AGENCY OPERATIONS 

In December of 1972, Ultrasystems contacted the National Office of 
Jobs for Progress* Inc. {Operation SER) to discuss the possibility 
that they» through their field offices* would participate in an 
experiment using the Cleff Job Matching System. As has been dis- 
cussed earlier In this report. It was at this time that the Office 
of Economic Opportuhity was reviewing the scope of the planned 
evaluation of the Cleff experiment in New Jersey. The idea of 
starting another experiment with a narrower set of objectives had 
been proposed by Ultrasystems and was being reviewed by OEO. OEO 
instructed Ultrasystems to begin the process of finding an organi- 
zation, involved in job placement* that would be willing to partici- 
pate in the experiment. The initial discussions with the SER 
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National Office led to individual discussions with the Project 
Directors of four SER offices in California. The four Project 
Directors and the National Office all agreed to participate. The 
initial planning for the experiment began after this agreement was 
obtained and was presented to OEO in January of 1973. The planned 
approach was accepted by OEO and Ultrasystems' original contract 
was modified to reflect this new task. 

Ultrasystems clearly stated to SER that the experiment was aimed 
solely at obtaining data that could be used to assess the effec- 
tiveness of the Cleff Job Matching System. Ultrasystems made it 
clear that it'could not and would not vouch for the system's 
validity in assisting SER staff in the job referral process. SER 
was interested in the experiment because they felt a strong need 
for better tools in executing their job placement function. The 
SER offices provide a wide range of services, all geared toward 
the eventual placement of the person in a job. The SER offices 
carefully screen applicants for enrollment in the training programs 
that they conduct. These training programs include the following 
types (the definitions are taken from the SER Managemoit Information 
System (MIS) Training Manual): 

a) JPC — Job Preparation Course . A job-related world-of- 
work program designed to prepare the enrollee to know how 
to seekt keept and exit from employment. 

b) ESL -* English As a Second Language . A language training 
class designed to assist the Spanish-speaking monolinguals 
to learn a survival job-related English that will prepare 
them for meaningful employment. 

c) ME (GEO) Advanced Adult Education (General Education 
Development) . A job-related course in advanced basic 
education designed to prepare the enrollee to enter higher 
skilled jobs and/or vocations. EnroUees whose functional 
education level is above the eighth grade will be con- 
sidered candidates for the AAE class. 

d) ABE (Adult Basic Education) . A course in basic education 
designed to prepare the enrollee to meet job-related gen- 
eral education prerequisites to certain jobs and/or 
vocations. EnroUees who function at the fifth grade 
level to the eighth grade will be considered candidates 
for the ABE class. 

e) OO r (On-the-Job Training) . The enrollee is being trained 
on the job by a private or public employer at his place 
of business. The employer Is reimbursed for the cost of 
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training by SER; the training Is done under contract for 
specific purposes and the employer Is firmly ccmntltted 
to retain the employee on a permanent t>as1s upon comple- 
tlon of the OJT contract. 

f) Vocation Training . A program Intended to provide voca- 

tlonal education and workshop training In clerical* service* 
semi 'Skilled to skilled occupations by an Instructor. 

SER also provides supportive services (legal* medical* transporta* 
tlon* and child care assistance)* and enrollees In the SER program 
are paid a stipend (from HOTA funds). Applicants tc SER are 
screened fbr enrollment In one or more of the training programs. The 
applicant screening process Is called A&O (Assessment and Observa- 
tion). The screening process Is to last no more than three days and 
"Is designed to facilitate the selection of those enrollees who are 
most likely to succeed." (Quote taken from the. SER MIS Manual.) 
As part of this screening process* one or more tests are ackninlstered. 
Generally* the "Wide Range Achievement Test" (copyrighted 1965 by 
Guidance Associates of Delaware* Inc.) was used. Other tests were 
also used. A sample listing of these follows: 

• SRA " Arithmetic Index 

• South- West em Publishing Company— Inventory Test- 
Business Mathematics for Colleges 

• Guidance Testing Associates— Test of Reading (R-2-0E) 
and the Spanish version (Pruba De Lectura). 

• SRA— Pictorial Reasoning Test 

• USES—GATB (administered by the USES) 

Applicants accepted by SER have an Employablllty Development Plan 
developed. As stated In the MIS Manual* this Is "a plan of action 
developed by the client* the counselor and the team (Employablllty 
Development team consisting of an Interviewer* job developer* and 
sometimes Instructors In addition to the client and the counselor)* 
which states in clear* concise terms the vocational goals* qualifi* 
cations needed for goals* employment opportunities* employment 
barriers needing attention* and the steps necessary to Implement 
the plan. This Includes training* remedial services* intermediate 
placement* and all supportive services necessary to implement the 
plan and enable the individual to enter a satisfactory job." 
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The primary reason Operation SER was willing to participate In 
the exper1,Dent was thel; dissatisfaction with the tests utilized 
In the screening of app* cants. This dissatisfaction was based on 
their feeling that most ^f the tests had ethnic biases and that they 
(the tests) did not seem to provide much assistance In the Job selec- 
tion/Identification process. It was the latter point that accounted 
for their willingness to participate In the Cleff experiment. 

As the above description Indicates, the operation of the SER field 
offices was In many ways similar to the operations of WIN and MOTA 
programs. At least the terminology and structure of their operations 
Is similar. SER Is, however, devoted to providing training, educa- 
tion, and placement opportunities to the Spanish-speaking American 
community. Whereas all enroltees of SER are not Spanish speaking, 
the vast majority are. The four offices that participated In the 
experiment were all very similar to each other In their operations. 
There were differences, such as the outstatloning of an £S staff 
person with the Job Bank listing In the Santa Ana office. Ift 
their basic operational structure, however, the four offices were 
essentially Identical. In addition, the National SER Office Imposes 
strict operational procedures regarding the completion of forms 
used to document and track services provided clients. 

This Management Information System Is Itself s1«n11ar to the WIN 
and MOTA reporting systems (and In today's manpower environment. 
It Is similar to the CETA MIS). These similarities In operations 
and reporting greatly facilitated the design of this experiment as 
compared to the New Jersey experiment with its vastly different 
agencies. 

Before turning our attention to the description of the experiment 
Itself, there are three other subjects that are worthy of mention. 
The first of these has to do with the rules established by SER 
for the conduct of the experiment. The SER offices all made it 
clear that they did not have the resources to administer SICLs. 
Therefore, a& part of the experiment, Ultrasystems employed a 
person of SER's choosing in each of the participating offices. 
These people were responsible for administering SICLs and tracking 
the clients. This idea of having a person in each office who was 
responsible for the administration of the system was in keeping 
with one of UltrJsystems* findings from the New Jersey experiment. 

The second subject has to do with an aspect of SER operations that 
was not discussed earlier. Whereas the major part of SER's opera- 
tions are Involved with training and placement of applicants en- 
rolled in their programs, th«y also provide direct job referral 
services to people not enrolled. In the terminology utilized, if 
a job referral made for an applicant not enrolled in SER programs 
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results 1n a placement* then this is referred to as a direct job 
placement, or a direct job entry. Correspondingly, enrollees 
placed Into jobs are referred to as Indirect job entries. In exe- 
cuting the experiment* no criteria were established regarding these 
two types of job placements. 

The final subject has to do with the history of Operation SER. In 
June 1966, two Spanish-speaking national organizations, LULAC 
(League of United Latin American Citizens) and the American G.I 
Forum joined to form Jobs for Progress. Inc. The program of pro- 
viding training, education* and placemeit services to Spanish-speaking 
Americans established by Jobs for Progress was given the name "Operation 
SER." SER, the Spanish verb for "to be" Is used as an acronym for 
"Service, Employment, Redevelopment." SER was funded by the United 
States Department of Labor, the Office of Economic Opportunity, and 
the Department of Health. Education, and Welfare. At the time of 
the experiment;. SER had field offices In 13 States, with the majority 
of these offices located In the Southwestern United States, I.e, 
California, Arizona, New Mexico, Texas, and Colorado. 



4.2 IMPLEMENTATION OF THE SER/CLEFF EXPERIMENT 

The Implementation of the SER/C'leff experiment consisted of six 
elements or areas of activity, as follows: 

Training In the administration and use of the CJMS 

Administration of SICLs to SER applicants and enrollees 
and the recording of placements made for these people 

Processing of the SICL and the establishment of the 
client part of the experimental data base 

Administration of JOCLs with the eniployers with whom 
formerly Sia'd SER clients were placed 

Tracking the retention of ',he clients who completed 
SICLs and whose jobs were JOCL'd 



It 1 



Analysis of the data 



Important to keep In mind that the experiment only sought to 
obtain the data needed to analyze the relationships between the 
Cileff system's person and job profiles and the eventual retention 

of the person on the Job. Therefore, the real purpose of the 
participation of the four SER offices was to provide people who 
would complete the SICLs and who would then be placed Into jobs. 
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4.2.1 



Training in the Administration and Use of the CJHS 



The initial training in the background, methodology, and use of 
the CJHS was provided by Mr, Michael Youchah of ADP-PDS Inc. 
This initial training session was held at each of the four SER 
offices and was attended by the staff of these offices. More 
intensive training was provided each of the people hired by 
Ultrasystems who were responsible for the administration of the 
SICls In each office. In addition, Mr. Ron Karris, who was a 
counselor at BICC, was hired for the duration of the experimental 
activities that took place in the SER offices. Mr. Harris devoted 
full time to assisting the SER office staff members, and especi- 
ally the SICL administrators, in the operation of the CJMS. 

Ultrasystems assembled a training manual for use of the SER office 
staff. This manual contained the instructional material prepared 
by Dr. Cleff {see Appendix E), and in addition contained 
the Job Cluster Register, the SICLs and JOCL booklets, and instruc- 
tions developed by Ultrasystems for Inputting the checklist answers 
and for reading the outputs. 

The iTKJSt important aspect of the training was provided by Mr. Ron 
Harris. One of the lessons learned from the New Jersey experiment 
was that one-shot training in the use of the Cleff system is not 
sufficient. There is a definite need during the system's initial 
usage for continuous guidance and assistance in dealing with the 
administration of the checklists and interpretation of the resultant 
scores. Mr. Harris also trained Ultrasystems and SER staff in the 
administration of JOCLs. 

The operation of the Cleff Job Matching ^stem was described in 
Vol. I of this report. In this description, Ultrasystems pointed 
out that in our opinion the instructional material available regard- 
ing the administration of the checklists and interpretation of the 
resultant scores does not adequately cover the real-world situations 
that arise. In particular, the issue of the degree to which the 
checklists are con^leted entirely by the respondent was discussed. 
The approach developed by BICC during the two years in which they 
utilized the CJMS was that one cannot always leave the respondent 
completely alone after the mechanics of completing the checklist 
is explained. There are no general rules regarding the manner in 
which the administrators oversee the checklist completion process. 
One essentially learns from experience the best way to handle these 
situations. It is important to keep in mind that the checklists 
are not a "test" in the sense that there is a correct answer to a 

Question. The basic issue has to do with the manner in which one 
eals with situations where clients (or on-the-job supervisors) 
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cannot decide on the phrases to choose or cannot find enough phrases 
to complete each page. This issue was discussed earlier. Suffice 
it to say that the manner In which the SICLs and JOCLs were adminis- 
tered during this experiment reflects in large degree the approaches 
developed and used by BICC and taught to SER 'and Ultrasystems by 
Mr. Harris. 

It should also be mentioned that there was one significant difference 
between the manner in which the SER/Cleff experiment administered 
SECLs and the approach used by BICC. In the SER/Cleff experiment, 
the SECL administrators were not SER counselors or interviewers. 
The administrators were specifically hired for this experiment. At 
BICC. the administrators and the counselors were the same p«jople. 
AT BICC, the standard approach was to interview the client before 
giving the SICL. The person interviewing the client and the person 
administering the SICLs were often or usually the same. The SER 
experiment did not operate this way, since obviously interviewing 
applicants was not Ultrasystems' responsibility. 



4*2.2 Administration of SICLs to SER Applicants and Enrollees 

Earlier we discussed that aspect of the experiment dealing with 
the selection of applicants who would take the SICLs. Our goal was 
to administer SICLs to all applicants and enrollees at the SER offices 
over the four-month time period. This goal was not achieved. The 
final decision was made by either the applicant or the training pro- 
gram instructor. People who came to the SER offices for direct 
placement assistance were briefly told about the system and asked 
to take the SICL, They obviously could refuse, SER itself does 
not administer the types of tests listed earlier to all applicants. 
Only those applicants who want to enter one or more of the training 
programs are given these tests. For the people who were currently 
enrolled in the training programs, the approach was different. The 
system was briefly explained to the instructors and their cooperation 
was solicited. With their approval, the SICLs were given to the 
entire class. Usually* this was done in two separate sessions- 
one for each of the two booklets. In a sense, the situation is much 
like what occurred in the New Jersey experiment. However, there 
was an important difference in that the SER experiment was not de- 
signed to provide comparative plac^nt performance data. 
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upon completion of the SICL booklets* the phrases selected by the 
person are entered Into the SICL Input form as shown 1n Figure 4-1. 
As explained In Vol. I, the SICL booklets consist of 
fifteen pages* with each page containing 16 activity phrases, I.e.* 
one for each dimension. The respondent chooses ten phrases on 
each page based on the rules governing this selection. The numbers 
of the phrases chosen are entered on the form shovm In Figure 4-1 
under the appropriate column headings governing the phrase seiec 
tlon criteria* I.e., most* least* more* and less. The respondent 
Is Identified by name* social security number, and/or birthdate. 
Each cllerit was assigned a unique five-dlgit number* the first 
digit of which identified the SER agency where the respondent was 
applying or was enrolled. The date the SICL was taken was also 
entered along with the name of the SER counselor or interviewer 
who was in charge of serving this Individual. The entries along 
the right-hand side of the SICL input form controlled the actions 
that would be taken with this input data. The first two boxes 
specified if the input was being submitted to correct a previous 
input or was a new 'input to be processed. The next set of boxes 
indicated whether the client was to be matched using only the prefer- 
ence score or the combined score or was to be matched in both ways 
(indicated by checking both boxes). The final set of entries indi- 
cated whether the match was to be made against the cluster profiles* 
active job profiles* inactive job profiles* or ar\y combination of 
the above. The last entry* labeled Spanish on the form* indicates 
if the client took the Spanish version of one'or both SICLs. 

The SICL input form was completed by the SICL administrator at the 
SER office. The input form was transmitted via facsimile to Ultra- 
systems' keypunch facility. A log of all SICLs administered and 
input was kept at each SER office and at Ultrasystems. Figure 4-2 
shows this SICL tally log. Each person who took the SICL booklets 
was identified in this tally log by assigned client number* name* 
social security number* and/or birthdate. The tally log also indi- 
cates whether the results were sent and received* whether the search 
was to be done using the preference (P) 2nd/or the combined (C) scores 
and whether the match was to be made against the cluster register (C)* 
active job profiles (AO)* or inactive job profiles (IJ). If the 
processing program determined that there were errors In input* then 
the column so marked was checked. A column for comments was also 
Included* as were columns for tracking the capture of the required 
SER MIS forms (to be explained later in this section). Msintaining 
tally logs at both the SER office and Ultrasystems enabled positive 
control to be exercised over the processing of SICLs. 

Figure 4-3 shows an exan^le of the output produced for a match using 
the combined scores against the cluster register. The client's 
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Figure 4-1 . SICL Input Form 
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Figure 4-2. SICL Input Tally Log 
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Figure 4-3. Sample CJHS Output Cluster Hatch 
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Figure 4-3 (continued) 
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Individual preference and experience dimension scores are displayed 
In the histogram and then summarized below the client Identification 
data. The client's occupational adjustment and difference Indices 
are given on ths line just below the client Identification data* 
The match scores to each cluster and the cluster dimension scores are 
then given In descending order < che match Index* Figure 4-4 shows 
a search using combined scores against the Inactive job profiles* 
It Is Important to note that no matches against active or Inactive 
job profiles were made during the four month period during which 
SICLs were given at the SER offices* As has been mentioned earlier* 
the completion of job profiles was done after the person who took the 
SICL was hired* The computer program was set up to enable one to 
match against job profiles * but the SER agencies only received cluster 
matches* The matching against the job profiles was done as part of 
the analysis of the data obtained* 

In addition to the tally logs* client case files were maintained at 
the SER offices and at Ultrasystems* The client case files at the 
SER offices contained the following: 

i) The SICL booklets completed by the client 

ii) The original copy of the SICL Input form 

111) A copy of the output generated 

1v) A copy the error messages {If any) generated as 
part of the processing and the SICL Input form used 
to correct these errors 

v) Copies of the forms completed by the SER staff and 
the client containing the client's socioeconomic data 
and the historical record of the services provided by 
SER and the outcomes of these services 

These client case files were established and maintained by the SICL 
administrators working in each SER office* These files were main- 
tained Independently of the client case files established by SER. 

The client case files maintained at Ultrasystems contained the 
following: 

1) The facsimile copy of the SICL Input form 

11) The original copy of the output generated 

111) The original error messages {if any) generated and 
the data sulwnltted to correct these errors 

IV ) The keypunch cards generated from the input form 

v) Keypunch cards generated by the processing program 
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Figure 4-4, Sample WMS Output: Job Profile Match 
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FILL CARoUsCL-tlirC C* HH|ac CS AFTCA AmHRCiNG liNlTS, ROUTlHC jOt 
IMM MM SO, I^AlN ST, LOS I^fCLtSt CS, flltS 
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Figure 4-4 (continued) 
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If at least ten error-free groups were not Input* 
then the program will not process the data. If there 
were, say, 15 groups Input but errors were detected 
In two of these groups (e.g., missing or duplicate 
phrase numbers), then the program will Identify the 
errors but will process the Input using the 13 good 
groups. Thus, the program will process 1nconq>1ete 
books and will process the two books cwnpleted by a 
given client when the number of good grouR$ per booklet 
Is not equal . 

The program linearly adjusts the resultant dimension 
scores to compensate for missing or unacceptable groups. 
Thus, If the client's preference dimension scores were 
calculated based on 14 groups, then the scores so obtained 
are adjusted by the ratio 15/14. 

Ill) The program checks the Inputs given In the boxes on 
the right-hand side of the form*. I.e., the entries 
which Indicate what Is to be done with the Input data. 
The program checks these entries, but If they are 
missing or Inconsistent, contains default logic v^lch 
sets the entries and processes the run. Thus, If the 
search on entries are both left blank, the program sets 
the entry to Indicate that the match Is to be made using 
the client's combined profile. If t!ie preference booklet 
was the only one Input, or If It Is the only one that 
contains at least ten good groups, then the entry Is 
reset to search on preference only. An error message 
Is printed to Indicate the problem with the Input entries, 
but the run will be made according to the default logic. 

Error messages are printed for each and every error detected, whether 
or not the Input was processed. Errors are corrected by Inputting 
the corrected or missing data using the SICL Input fom. Errors 
can be corrected with no new output being produced. I.e., entering 
the client's birthdate when this was the only Item missing, but 
Indicating that a new search Is not required. 



4.2.3.2 The SER Management Informat1on_System 

The SER Management Infomatlon System (HIS) consists of five forms 
and four monthly reports. As has been mentlonedy the system Is 
similar to the WIN, MOTA, and CETA reporting systems. The five 
reporting forms are as follows: 
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S-1 Applicant Infonnation Record (see Figure 4-5). 
This fom is used to collect pertinent Information on 
the personal, educational* and occupational character- 
istics of each individual who applies to SER for services. 
For those individuals who come to SER for direct job 
referral services, this is the only form used. The job 
referrals made on their behalf are recorded on this form 
as is the one-month retention foUowup required for all 
direct job placements. This form is partially completed 
by the applicant. 

S-2 Employability Development Plan fEDPl (see Figure 4-6), 
After an applicant is enrolled into the SER program for 
training, an Employability Development Plan (EDP) is deter- 
mined and drawn up on the S-2. This includes the vocational 
goals, test results, qualifications needed for goal, the 
employment opportunities, the steps necessary to reach goal, 
and other circumstances affecting job entry possibilities 
that need attention. In the EDP, every status that the 
enrollee will be in is written down. For each status, the 
EDT mend)er responsible, the starting date, the training 
days to be spent, and, if change is required, the reason 
for modification is also written down. 

S-3A Status Change Notice (see Figure 4-7). 
The S-3A provides a record of an enroUee's movement through 
the SER program and is the basis for tallying and checking 
program activities for monthly reporting purposes. The S-3A 
is filled out by the team each time an enroUee's status in 
the program changes. That is, it is filled out whenever an 
enrollee enters the program, moves from one component to 
another, moves iiito or mt of a holding status or job entry 
status, or terminates from the program. A different S-3A 
is filled out every time the enrollee changes status. The 
S'3A is also used as a memo of the next probable status 
change. This S-3A is referred to as the next S-3A. 

S-4 Enrollee Record (see Figure 4-8) 
The S-4 provides a chronological history of the enroUee's 
activities in the SER program and a record of the time &nd 
cost that the enrollee spent in each component and status. 
The S-4 is used to record all of the enroUee's movements 
through the program. It is a summary record on one piece 
of paper of all the enroUee's program activities: job entry 
and follow-through information. The S-4 is updated from the 
S-3A forms that are filled out each time there is a change in 
the enrol lee's status. 
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S^l APPLICANT INFORMATION RECORO 



1 NAMf (I 4t|,F»t1,Mjcld4«| 



7. SOCIAL SECIiPllTV NO. 



4. TEuei»HONe NO. 



LENGTH IN 
IMMEOIATE 
AR£A 



UJS. CITIZEN OR 
■tCRMANENT RESIDCNT 
1. YES 



.HO 



fSPtClly) 



7. NO. In FAMILY 
{tncludini Vour««tt) 

No. Ol DC^NOCNTS 



I estimateo total income 

LAST »2 MONTH* 
API>LICANT I 

Family i 



9. HEAOOF FAMILY 
OR HOU5EHOLO 
I. „ VEI 
2 >-NO 



10 DATE OF BIRTH 

Mo Oay YEAfl 
AOE_ . — 



U.SEX 

I. ^Male 
2 Female 



12. ETHNIC (ChacKOn*) 

1. M«j(lGM-Am«rlGin 2 CuDin 3. Pu*rlO Rl»n 4. AltO-Am»ttGiA 

5 Anplo.AmftrlGin < Amtrlcan Indian 7. ^ Olh«r {S»«Glfv) 

Oit«t Nami 4 locahon UHv wtt tUl*) ol lati Khnol atUnflid 



i. EOUCATION 
fA) Old ynu 4raduiU from HtS^i School or r«<«lv«a G.E.a? 

J. ^YES ... Oif«<month/y«ir| 2* NO 

Htf*itil Gridt CompitUd llnclud* Col^af ) i 0*t'**(i)- 



t4 MtL^TAftV SERVICE STATUS 
t V«t«ran . OtKhirf* Oil* 

2^ . Oth«r OrallClatt.. 

<Sp#Ct'y) 



t5 HANOICAP^D 
|.~VES 
tS»«Glfy) 



U. PUtLtC ASSIST. RECIPIENT OR 
UNEMPLOYMENT INSURANCE CLAIMANT 

1. ves. 



2. NO 



{S»«Gify) 



XT. MARITAL STATUS 
1. . StntI* 2. Marritd 

3. „ S«Par«1«d 4. . Oivnfctd 
5. . - Wjdnw«d 



I PARTtCtPATtON IN OTHER FEDERAL PAOGRAMS<SP«Ctfy I 



Oit* LKt PirtlctlMttd- 



19. Employment STATUS 

1. ^^EmPIOYtd 2 Und«r«mpioy*tf 3. . Un#moioy«d 

WEEKS UNEMPLOYED 

iM 12 MonlM Curttnt Sp«lt (Ummpioytd omy) ^ 



20 EMPLOYMENT HISTORY (Start with prt»«nl tob. H »mPl0ytdf and wofK b*GK) 



FROM TO 



EMPLOYER NAME ANOAOORESS 



jOt TITLE 



'^^^^snp'Y^ ]^E;:sofrFbin:EAviNo 



21 LIST OTHER SKILLS AND ABILITIES (Includ* Oth«r •Mp*ri«nG« or Uilnini rMtlvttf) 



22. VOCATIONAL GOALS (SPtClly) 
Train rm OWlrtd 

OCGUPillon 0«tlr*d 

Stla^y D»llr*d 



I CERTIFY THAT ALL OF THE AftOVC STATEMENTS ARE TRUE 
AND ANY FALSIFICATION WILL BE GROUNDS FOR TERMINATION 
FROM THE.S£S PROGRAM 



APPLICANT SIGNATURE: 

X 



DATE* 



INTERVIEWER SIGNATURE. 



" too NOT WRITE iZLOW THIS LIN£) 


23 MEENTdY 

I. VES 

2 No 


Z4 MONO 
LINGUAL 

L YES 

2 NO 


25 MIGRANT 

1 Yes 

2. NO 


2%. CARRIERS TO EMPLOYMENT fSp«Glfy) 

J.»ME0ICAI 3. _ LEGAL OTHER ^ 



27 DtSPOStTlON or APPLICANT {Chtckl 
J. » OiMdvinlaf«d 
2. _ Ccrllltid (ttvWhom)^ 



S. _ RMlwd SupporHv* S4rvlCt« tSp«Glty)- 



4 



Entllltd Ttalnlni Allowanc* Ptt W««K 

Regular I — — SubtitttAC* I 

Trantt>OtUtlon TOlat i 

. Enrontd <Oit«) 



< Jot> Cl4f«ltl«d 



JOD Tit It 



DOT Cod* 



GU£llfl«r 



Itt- 



I Olhtr (SPKIfyl 



Znd- 
Ird 



7 — Rtfritd ^o Job fComPitl* M*m £SI 



21. JOe REFERRAL ANO JOB ENTRY INFORMATION 


DATE 
REFERRED 


EMPLOYER NAME ANO ADDRESS 


TELEPHONE 
NUMBER 


JOB TITLE 
&O.T. CODE 


HOURS PER 
WEEK 


HOURLY 
WAGE 


DATE 
ENTERED 













































?9 



Contact Date 

DatEPerformED- 

PERFORMEDtY 



RESULT I 

I Employed 

2, Not Cmployeo 

1. „ Not contacted 



COMMENTS: 



Person Contacted- 



10 SUPPLEMENT INFORMATION (UM t>KK M tj ^ copj^OrHy tor addltlOn«l IntOfmitlOnJ 



Figure 4-5» SER Applicant Information Record Form (SI Form) 
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@ S-2 EMPLOYAIILITY DEVaOPMENT PLAN 



t NAME (USI. First, Middle) 



2. SO, SEC. NO. 



3. DATE ENROLLED 



4, DATE PREPARED 



I VOCATIONAL GOALS 

FIELD OF WORK 

SPECIFIC OC JUPATION 

ALTERNATIVES (Optional). 



iIh test results 

MEDICAL 

FUNCTIONAL LEVEL 

API ITUDE/PROFICIENCV 

OTHER (Specify) 

TESTS USED 

III QUALIFICATIONS NEEDED FOR GOAL (SV 



IV, EMPLOYMENT OPPORTUNITIES. 



V. STEPS TO REACH GOAL 



PROGRAM COMPONENT/STATUS PARTtCIPATlON 
(Numb«r Chronol09<Ciriy) 



EOT MEMBER 
RESPONSIBLE 



STARTINQ 
DATE 



TRAINING OAVS 
TO Bt SPENT 



REASON FOR 
MODIFtCATtON 



VU OTHER CIRCUMSTANCES AFFECTINGJOB ENTRY POSSIBILITIES THAT NEED ATTENTION 



Figure 4-6. SER Employability Development PUn; $2 Fom 
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I4A* STAlUt CHANM NOflCI 



1. i"t 



>. SOC. SIC, NO. 



i, OATI(M»CHAHH 



NEW ENROLLEE 
RE-ENTRV 

4. LEAVING COMPONENTATATUS s. ENICRINB 



n *• 


ESL 






ABE 

AAE fOEO) (SffMifv) 






VOCATIONAL TRAINING 
(CodHMt* Ittm •) 






0 JT 
(Compitt* Itwn •) 






PLANNED SUSPENSE 
{Complft* Ittm t) 




1 II 1 


JOB ENTRY HOLOINO H. | | 
PROGRAM HOLOINO (SpMtfy) 1, 



□ 



TMMIHATION 

130 INOIRECTJOiENTRV 
(Cofliplfft* Itwm 6 7) 
Annual Wsf« at EnrvtliiMnt $ 

1*0 NEUTRAL TERMINATION (SP«lfy) |~] 
190 OROPOUT {SpiClM |~] 
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7. JOi ENTRV INFORMATION 



No 



AMrtM- 



Clty. State,. 
Hrf,/Wk. 



Hourly WsfO- 



,Z\p — 



Job TItli. 



OOT Codo teOlfft). 



r TftAtNINO INFORMATION (OuUldiSEft^o(«t) 
Trtlnor 

T#l#phofM N^- 1 — 



city, SUt«,. 
Tralnlni 



-Zip. 



. Tri»|. Allo«wnct/Wk. 



OOTCotft (6 Offit) 

OJT C«ntr«ct: Fr«m- 



9. STATUS CHANGE VERIFlEO BVi 



10, NEXT PROBAiLE STATUS CHANGE 



11, TOTAL PARTICIPATION DURING LAST STATUS 



12. TOTAL TRAINING ALLOWANCE DURING LAST STATUS 

13, rrtuuPHTK, 



Figure 4-7, SER Status Change Form: 
S3A r or m 
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M ENKOiUI MCOKD 



\ NAME (Uii FiMi MMIt) 



r SOCIAL SECURITY NO. 



4* PHONI NO. 



I. DATE OF CHANGE 



6. TRAINING ALLOWANCE PER WEEK 



Tr«niport«tlcn • ' 



TottI • 



PROGRAM COMPONENT/STATUS PARTICIPATION 



1 COMP<y<tMt/itJ^tUi 






* fOIAL 




OA 




MV| 






tn 


































































































- 



































































































TERMINATIONS 



L REASON FOU 
TEKM INATiDN 



INDIIlfCT iOi tNTKY 



NtutlAL iip4t\h^ 



r CHECK 



DATE or 
TttMlNATIOH 



9. e.D.P. COMPLETED (Indlffct Job Entry only> 
p Ytt P NO 



ERiC 
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10. JOB ENTRY INFORMATION 



EMPLOYEIL 



TELEPHONE NO.. 
ADMESS 



CITY. STATI_ 



1\K 



HOURS/WIL 
JOI TITLi^ 



HOURLY WA«E. 



DOT CODE U DIfltL 



tl. TRAINING INFORMATION (OutsidftSER Projtett 
TRAINER 



TELEPHONE NO.. 
ADDRESS 



CfTY, STATE. 



ZIP. 



TRNGT 

HRS/WK ALLOWANCE/WK, 



TRAINING. 



OCT CODE (6 OmU 



12. focloW'Thr6uqh information 



1 MOMTH 


Z MONTHS 


3 MONTHS 




fit 


NC 


t 


Ht 


HC 


E 




HC 





















• MONTHS 





NC 


HC 


UPQRAO 


•IHCR. 













13. COMMENTS 



Figure 4-8, 



SER Enroll) 
S4 Form 



Record: 
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• S-5 Follow-through Record (see Figure 4-9) 



This fom is used to record the six -month retention 
followup that is required for all SEP enroUees placed 
into jobs. This form is initiated when the enrollee is 
placed into a job or is placed via an OJT contract. 

Copies of the S-1, S-4. and S-5 were to be obtained from the SER office 
for each client whu took the SICL. The retention followup data 
used in the analysis was obtained by Ultrasystems directly and was 
not obtained from the S-5 form. The S-5 forms filled out by SER 
for all the enroUees who took S!CLs were not obtained and so no 
comparison of their foUowup data with that obtained by UUrasystems 
is available. It should be note^* that the retention of the direct 
job placements is only determined by SER for one month. Copies of 
these foms were obtained only while the SICL administrators were 
working at the SER offices. After the four-month period of SICL 
administration ended* the subsequent job placement data was obtained 
via periodic visits to each of the SER offices. During these visits, 
UUrasystems wenr through the SER client case files for those clients 
who took SICLs. Once UUrasystems attained the required number of 
job placements for the job profiling and retention foUowup activi- 
ties* the process of searching the SER case files was discontinued. 
Thus UUrasystems did not obtain data on the services provided all 
the clients who took SICLs. 



4.2.3.3 Special K^punch Cards Generated by the CJMS 

Computer Program Utilized in the SER Experiment 

In order to facilitate the eventual analyses, a special set of com- 
puter-readable output cards was generated for each client who took 
the SICL booklets. These output cards contained data that Identified 
each client and contained the client's preference, experience, and 
combined dimension scores. These cards could then be input to the 
program to generate matches against the job profiles that were later 
obtained without going through the processing associated with the 
conversion of the input SICL phrase numbers into dimension scores. 



4.2.4 Administration of the Job Outline Checklists fJOCLs) 

The Cleff system job profiles, for those jobs into which SER clients 
who took SICLs were placed, were obtained by UUrasystems. These 
profiles were obtained after the client was hired and. In some cases, 
after the client had already terminated. AH the profiles were 
obtained in the summer and fall of 1973. The data obtained from 
the SER client case files, for those people who took SICLs, provided 
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S*S FOUOW'THROUGH RICORD tEv 



1. NAME 4v4«t. RmK UMk\ 2. SOC. SEC. NO. 


ADMESS iUumUt, SirMt. Ctlv. Stit«. C«J«» 4- TELEPHONE NO. 


5. JOI ENTRY INFORMATION 

Em^Uv*' . . - . Convict - * . _ 

A^JJiAii ^ 7Jp CmAm. TaUphon* Num^tf 

Jsk Tl*l. n^T r«u U QiMttI 


H«uri WM(t . Hw\i W«9« ^ 0»U of J^b Entry . 


4. CONTACT DATES 

1 Month 2 Mnrtiht a M"-T*»« ^ u«ii*hi 


1 MOKTH 


2 MONTHS 


lt««Jt Of CMt«d IChsclt) 

□ Em^UyW □ Not Employed 

□ Not Contocltd ISpoctfyt 

Noir Job TitU. . OQT (4) 

H%w Em^Uyor-. ^ ^ „ _ ^ , . _ 

How HouHy Wo»o, Hrs,/Wfc. 


lt««ult Of CtKbd <Ch«cU 

□ Em^Uytd n Not Em^Uv«J 

□ N^t C«nt«ctW <SMc<fv» 

N*w Jolt TitU DOT U) - 

H^w Emilov«f • - „, 

AdJr«M — _ >^aii*^_^_ , ,. ^ . 

H%w Hwrly W»»t^ « , , « Hri,/Wk^ 






3 MONTHS 

CVM* PufforiMd _ ^ fty 

P«rion CorttKM ^ 


« MONTHS 

D«U ^fofnmt . , _ ^ . . 

Ccmti&lod 


lt««ult Of C*Kt«ct ICkiclt) 

n Em^l*vW n N«t Employed 
□ N«t C«nt«ct«J 1Sp«dfy) 


Reiult Of Contod IChocU 

□ Em^ov^il □ Not Em^loyod 

□ Nit Contoct^id (Spodfyt 


N««r J«l> Tit;»- , QOT UL 

Ntir EAipUy«t . _ _ 

AJdrwi _ n«n« - ^ . 
H%w H«utly W«9« Hti WL 


Now Job Tttio DOT U) . 

Mow Employor , ^ _ _ , _^ ^ 

Addroit ^ ^hono. 

Now H^rlv Wtft Hn, WL. - 



Figure 4-9, SER Follow-Through Record: 
Form S5 
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' UUrasysteins with the InforrMtlon needed to Identify the jobs 

and the employers with whom the experimental group clients had 
been placed. UUrasystems contacted these employers directly* 
by phone* and asked them If they would allow us to administer 
the JOCL with the Immediate job supervisor of the person who 
had been hired. The purpose of the request was explained and a 

> pledge of confltfentlallty was given. In some cases* the employer 

requested a letter from UUrasystems again explaining the purpose 
of the request and pledging, In writing* confidentiality of the 
Information obtained. Exhibit 4-1 shows such a letter and the 
pledge of confidentiality. Approximately 25 percent of the 
employers contacted refused to allow us to administer the JOCL. 
In addition, six employers who agreed to let us administer JOCLs 

, failed to finish the checklist. The final data base consists of 

110 Individual JOCLs administered at 100 separate employers^ 
covering 142 clients. 

' The administration of the JOCL consisted of the following steps: 

1) Verification that the client recorded by SER as having 
been placed at the company was Indeed working there or 
had worked there. This Included verifying the hire 
date and the job. No cases were encountered where the 
person recorded by SER had not been hired by the company. 
(Note: Several of the employers who refused to allow 
us to administer the job outline stated that the person 
whom we named never worked there. It Is UUrasystems' 
opinion that this was their way of saying that they did 
not want to be bothered.) 

11) The oi-the-job supervisor who was to complete the JOCL 
was asked to bHefly describe the job that was being 
profiled. In many cases* the UUrasystems staff person 
administering the JOCL asked to be* and was* shown people 
doing the job. In some cases* this was not necessary 
because the job was one that was familiar to the Ultra- 
systems staff person, e.g., a bank teller, a parking lot 
attendant. 

Ill) The mechanics of completing the JOCL were explained 
using the terminology Included In the JOCL booklet. 



1v) The finished booklets were checked for completeness and 
In some cases the resultant profile was drawn and dis- 
cussed with the respondent. 



ERIC 
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EXHIBIT 4-1 



12 Ouly 1973 



Los AngeleS} California 90005 
Dear Mr. SHHi 

On Tuesday and Wednesday, July 10th and Uth, Ms. 
from Ultrasystems, Inc. spoke with you concerning the possibility 
of arranging an Interview with your company with regard to a 
government- sponsored research effort that Ultrasystems is conduct- 
ing. Ms. Informed me that you were interested in 
agreeing to the Interview, but that you would like some more 
Information regarding the nature of the Interview arid the research 
effort Ultrasystems is engaged In. This letter will briefly 
explain what the requested Interview Is all about. 

The research effort Is being perfonned by Ul -rasystems. Inc. under 
contract to the U.S. Office of Economic Opportunity. The research 
concerns evaluating the effectiveness of an experliinental computerized 
man-job matching system. The purpose of this project Is to try to 
develop aids foi^ use In screening applicants to find the workers who 
best meet an employer's job requirements. As part of this research, 
Ultrasystems, Inc. has been working with the Los Angeles SER office. 
For the last few months -the Los Angeles SER office has been adminis- 
tering a set of multiple-choice questionnaires to the people who come 
to their office for job placement and training assistance. Their 
records indicate that, to date, two of the people who completed these 
questionnaires were hired by the B^^^HHHHHH^^^HIHBH 
The requested visit to your company is for the purpose of Interviewing 
someone In your organization concerning the nature of the jobs that 
these Individual s were hired Into. (The Iridividuals and their respec- 
tlve jobs are: i Mi^M who was hired as a Maintenance Man, and 
MHHMHB;, who was hired as a Jr. Clerk (Small Office), as 
Indicated by our records). The interview involves the c(»nplet1on of 
a structured multiple-choice questionnaire that defines the particular 
job.' I have attached a copy of one page of the ten page questionnaire. 
As you can see, the questionnaire Is fairly simple and straightforward. 
It Is Important that the person completing the questionnaire 1$ very 
familiar with the particular job in question. 



o ^HVwniMS. IN<*. 
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EXHIBIT 4-1 (continued) 



12 July 1973 
Page 2> 



The information obtained from your organization through this 
questionnaire will be treated with complete confidentiality. Your 
name* the company name* the employee's natRe* and the name(s) of 
the people Involved in completing the questionnaire will never be 
revealed.. I have enclosed a letter confirming our pledge of 
confidentiality. 

I hope that this brief description will help clarify what the 
purpose and extent of the interview is. I also hope that you will 
pemit us to complete the interview, at your convenience. 
Hs. will contact you again in the near future regarding 

this request. 

Sincerely yours, 
ULTRASYSTEMS, I(1C.' 
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EXHIBIT 4-1 '{continued) 



12 July 1973 




Los Angeles, California 90005 



Dear Kr. IHHH 

Ultrasystetns hereby certifies that all the Information provided 
to Ultrasystms regarding job descriptions obtained in conjunction 
with Ultrasystems' government- sponsored research effort will be 
treated In a confidential manner. Ultrasystems specifically 
certifies that the name of your organization, the names of all 
personnel Interviewed, and all applicants and employees for which 
data has been obtained, will not be used In any reports produced 
by Ultrasystems. Thus, the data provided by your organization will 
be used In a confidential manner to assess the effectiveness of the 
Oob Matching System being Investigated. 

Thank you for your assistance to our staff In Its activities 
related to the Job Matching System evaluation undertaken under 
Contract No. B2G-5415 for the United Statr-. Office of Economic 
Opportunity. 

Very truly yours. 



ULTRASYSTEMS, INC. 



v) The respondent was told about the purpose of the study 
and their cooperation 1n the retention followup phase 
was solicited. Only one employer was later to refuse 
to provide retention followup data. 

Host aspects of JOCL administration are straightforward. Howeveri 
one does encounter situations where the respondent cannot choose 
enough phrases on each page. ThiF Is often due to the respondent falling 
to find phrases that directly relate to the job being profiled. 
Ultrasystems followed the approach taught to us by Mr. Ron Harris, This 
consisted of repetition of the Instructions emphasizing the Instruc- 
tional phrases most resemble , most like , etc. If this failed, then 
we used examples based on the phrases In the booklet but related to 
a different job. In other words, we would briefly describe another 
job and then show how the phrases listed were like or resembled 
behaviors required for this other job. In a few cases, this was 
not enough and these resulted In the respondent reVusIng to complete 
the booklet. 

There Is one other aspect of JOCL admlnlstra+.lon that needs to be 
mentioned. This has to do with the card sor£ portion of the JOCL. 
The "official" Instructions regarding this aspect of JXL administra- 
tion were briefly given in Section 2 of Vol. I and are shown In 
Appendix E. Ultrasystems, with the agreement of ADP-PDS Inc., did 
not administer this portion of the JOCL exactly according to these 
Instructions. The difference was not In the manner In which the 
cards were chosen or discarded. The difference was In the manner In 
which the cards were scored. Ultrasystems had the respondent rank- 
order the two sets of five cards. Thus, for the five cards chosen 
which best describe what must be done on the job In order to do It 
right, the respondent rank-ordered these cards Instead of Independently 
assigning the scale values defined as outstanding, very great, great, 
a lot, and some. Thus, only one card would be listed as outstanding, 
i.e., the highest ranked card. The "official" rules of JOCL ad- 
ministration require that the final sum of the positive and negative 
scores across the sixteen dimensions add to zero. Independently 
assigning scale values to each card means that the sum obtained may 
not add to zero. One then has to "nonnalize" the Individual dimen- 
sion scores. No "official" Instructions were ever provided Ultra- 
systems dealing with how one goes about normalizing the dimension 
scores. The only Instructions givet. us were that one adjusts the 
highest positive and the lowest negative scores inward (toward 2ero) 
so that the sum balances. This reduces the peaks of the resultant 
profile. Instead of doing this, Ultrasystems elected to rank order 
the cards and was told by ADP-PDS that this approach would be 
accep ^ble. 
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Before tnoving on to the next topic, there is one other subject to 
be discussed* This has to do with the issue of non-response bias 
due to the failure to obtain job profiles from all employers con- 
tacted and, in a sense, due to the selection of both the clients 
who took SICLs and those included in the foUowup data base* As has 
been discussed, every applicant and enroUee at each SER office did 
not take an SICL and, of those that did and were placed, only 142 
were included in the data base. 

Thus, we have a situation where the selection of clients to take 
the SICLs was not complete and was not random* In fact, the process 
was one of self-selection where the self refers to the client or 
to the instructor* The inclusion of clients in the data base was 
based on their hire date; inclusion was done sequentially as opposed 
to, saty, randomly selecting clients from all those hired over some 
time period. Failure to obtain a JOCL meant that this client was 
replaced by another client farther down in the hiring date queue* 
Failure to subsequently obtain retention data meant that the client 
was dropped without replacement* 

Because the analysis of the retention and Cleff score data was not 
done with the objectives of: 1) comparing this data to another set 
of clients who did not take the SICL (a comparison group), and 2) 
obtaining representative retention data about SER placements, Ultra- 
syatems believes that the manner in which the data base was assembled 
does not bias the results. As will be shown in Section 5 to follow, 
the client characteristics (of those clients included in the data 
base) were broadly representative of SER clients in general* Whereas 
no comparative job characteristic data was obtained, the spectrum 
of jobs that were profiled was quite broad and was essentially com* 
pletely in the areas generally characterized as low to semi-skilled* 
The key point is that the analysis was based on the relationship 
between the Cleff scores and the client's eventual retention. Since 
no cut-off score criteria were imposed, the resultant score distribu- 
tion was quite broad* 

In regard to this issue, it should be noted that no distinction has 
been made between direct and indirect job placements* Again, the 
analysis was not done with the purpose of analyzing the comparative 
retention of people trained by SER and people who were only provided 
job referral services by SER* 

There H no question, however, that the clients whose data were 
utilized are not broadly representative of the population of low 
or semi-skilled job seekers*. The vast majority of the clients were 
Spanish-speaking Americans* The initial development of the CJMS was 
done using a totally different type of person, i*e., not a Spanish- 
speaking American (see Appendices A and B)* What effect the type of 
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client (defined by more than just the Identification as a Spanish- 
speaking American) has on the resultant SICL profiles and there- 
fore on the resultant job-cHent match scores Is not known. No 
statistical analysis of client characteristics and SICL profiles 
was done under this study. 



4.2.5 Retention Tracking 

The tracking of client retention was done at four time points, 
i.e., December 31, 1973, February 28, 1974, Hay 31, 1974, and 
December 10, 1974. In addition, some termination data was obtained 
at the time the JOCL was administered} i.e. , some clients had already 
terminated. Except for these cases, all the retention followup was 
done via telephone conversation with the employer. The employment 
status of the individual involved was determined and, if terminated, 
the date of the termination (last day worked) and the reason for 
termination were obtained. Once an employer indicated that the 
person had terminated, no more followup regarding this individual 
was done. 

Ultrasystems did not contact any clients regarding their employment 
history. In addition, eo^loyers were not asked about job changes 
relating to the individuals in our sample; i.e., the retention was 
based on their en^loyment with the employer. This is a serious 
issue in that the match scores are based on the profile of one job, 
i.e., the original job. 

As was mentioned earlier in this report, there were cases v^ere a 
given employer changed their employment status answer from one 
followup call to the next. This happened when the employer indicated 
that the person was still working and then at the next followup 
time point indicated that the person had been terminated and the 
date given was prior to the earlier call. When questioned about 
this, the employers indicated that this was due to the manner in 
which their payroll records were processed and that at the time 
of the followup call the only records available Indicated that the 
person was still employed. When these records were updated, the 
new answer was available and this was obtained at the next retention 
followup time. This occurred in two cases between the December 31, 
1973 and February 28, 1974 time points. 



4.3 SOME OBSERVATIONS OF ULTRASYSTEMS' STAFF REGARDING 

THE SER/CLEFF EXPERIMENT 

Exhibits 4-2 and 4-3 present verbatim written observations of two 
of the staff people involved in this study. The first (Exhibit 4-2) 
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was prepared by Hr. Ron Harris at the conclusion of his activities 
In the SER offices. Hr. Harris* as has been mentioned previously* 
had been a counselor at the Business and Industrial Coordinating 
Council In Newark* New Jersey for approximately two years. He had 
been extensively Involved with the CJMS both prior to and during 
the New Jersey experiment. Mr, Harris provided continuous on-site 
guidance and Instruction during the course of the SER/Cleff experi- 
ment. The second set of observations was prepared by firs. Melinda 
Green* who was one of the people hired to administer SICLs at one 
of the SER offices, Ultrasystems has elected to remove references 
identifying particular SER agencies and their staffs from these 
observations. The full text of these observations is presented 
with the references blocked put. 

Ultrasystems believes these observations are easily understood 
and address Issues discussed In this report. Therefore* no further 
comment will be given. 
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EXHIBIT 4-2 



INTIKOFFtCC COKKISrONDENCE 

TO: CC. DATE; 12 July 1973 

SUBJECT: RaiarKs on CJMS from: - 



Undertaking Option "B" of the O.E.O. Cleff Job Hatching System 
evaluation as proposed by Ultrasystems has broadened the undersigned's 
perspectives in terms of the systems acceptance, its capacity and its 
feasibility as a mechanism to arrest underemplo^ent and unemplo^nt. 
Essentially, it is these new perspectives that will be discussed in 
this comnunlcation. 

Working with the SER organization at various local offices has brought 
about differing responses to the system. I'd like to spend time out- 
lining the involvement and reaction of each office individually, and 
perhaps concluding with reference to the overall effect of the Cleff 
Systan as related to the SER organization. 



Implementation of the systan was primarily in conjunction with the 
scheduled procedure as outlined in interoffice memorandum from 
dated 3 April 1973, Subject: Objectives of CJMS Mini -Experiment. By 
the week of April 2, 1973, CJMS technician was fairly comfortable In 
assuming SICL administration procedures. During week of April 16, I 
was well into instructing the counselors on interpretation of computer 
printouts and analysis of SICL/client relationships. At this level, 
reaction and response wsre positive. Sr. counselor at this office 
opened discussion for utilization of the Cleff System as part of their 
normal intake procedures. There was some apprehension on the part of 
one subordinate counselor at this point. This, however, is attributed 
to this counselor's absence from two consecutive training sessions and 
the consequent lack or knowledge as to the system's functional design. 
Scheduling at this point pr(»npted JOCL introduction. The subordinate 
counselor's problems were resolved through supplemental instruction 
from the Sr. Counselor. 

Soliciting participation and cooperation of job developers and class 
instructors as related to their proposed involvement with OOCL pro- 
cedure proved somewhat rougher than anticipated. Then^current work 
loads bore heavily on their schedules. Availability for training was 
minimal, necessitating a meeting with the Executive Director with 
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EXHIBIT 4*2 (continued) 



U July 1973 
Page 2 




Continued) 



efforts directed at achieving participation of job developers as 
proposed In JOCL scheme. Participation was achieved, but attitudes 
were not particularly conducive to effective response to JOCL 
Instruction. I think our problem here was that aiv Involvement on 
their part was In addition to their alreacfy heavy schedule. 

The decision to continue on our own was good. I feel we would have 
had more difficulty achieving the needed data for the evaluation had 
we continued their Involvement. The response of business and Industry 
In this area was generally good. Explanation of the system's design 
and Intent brought mny favorable, positive reactions. All will be 
looking for follow-up Information from Ultrasystems/SER. Some 
Inquired as to how they would or could benefit from the system If 
the evaluation shows positive proof. I think that for the most part, 
their Involvement In a computerized matching system Is new, and 
they are interested. You should get a high enough percentage of JOCL's 
to validate the match analysis. 



This office, much smaller In size, and consequently volume, moved 
smoothly through SICL Introduction, administration. Instruction, 
counselor Interpretation, and client analysis. The CJMS technician 
assigned to this office, conscientious In her procedure and the scope 
of her responsibility, performed SICL administration and clerically 
related assignments efficiently. Initial apprehension was prevalent 
during counselor Introduction to SICL Interpretation. Subsequent 
training. In addition to sample case match analysis demonstration, 
alleviated crucial skepticism. JOCL Introduction again brought about 
schedule conflicts. Job developers were too burdened with routine 
responsibilities to participate effectively In JOCL administration and 
Interpretation. Vacation schedules, monthly quota requirements, seminars, 
etc. were among reasons for job developer non-avalUblllty. As decreed 
by the Executive Director of this office, SER staff members would be 
unable to participate In JOCL collection due to ths extent of their 
ongoing workload. Ul trasystems personnel were thus provided open 
avenues to persue and obtain JOCL's. The SER staff has been Instructed 
to assist Ultrasystems personnel by providing Information relative to 
those firms by whan SER clientele have been hired, to Include names 
and positions of contact personnel, and suggested approach. Limited 
assistance In establishing Initial contact will be available through 
job developers If deemed necessary by Ultrasystems. 
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EXHIBIT 4*2 (continued) 



12 July 1973 
Page 3 




Some Initial problems experienced here. CJMS technician experienced 
difficulty In Implementing SICL procedures effectively. This 
difficulty v»s subsequently attributed tc a lack of a peopl e*or1 ented 
background. Much more additional Instruction, guidance, and supervision 
was required at this office to bring technician to a comfortable level, 
consuming time and requiring sch(idul1ng revisions. Technician's 
attitude and educablllty were beyond reproach. Lack of exposure to 
manpower evaluation procedures and management, and/or supervisory 
techniques on the part of the technician prohibited schedule maintenance 
and required delays In the Introduction of the system to counselors. 
This office additionally experienced Inopportune job title changes, 
resulting In one counselor with an overbearlna case load, and one 
"trainee" then currently being Indoctrinated in counseling techniques 
and In 1n*house procedures at this office. While there were signs of 
latent Interest and concern for the system and Its potential, very 
little time was available to the counselor(s) to effectively engender 
the system's capabilities and capacity as a job matching mechanism and/or 
a counselor tool. Meetings with supervisory personnel of this office 
detailed Intake processing procedures that conflicted with SICL adminis- 
tration to walk-In personnel by our technician. This conflict 
eventually precluded further administration of SICL's to walk-Ins at 
this office, limiting our SICL Intake to an Independent request bdsis 
only, reducing the volume considerably. No true resolution came to 
fruition, severing SICL accumulation. Job developers spoke negatively 
about JOCL acquisition. In all, the system didn't seem to have 
positively Influenced this office. Negativism prevailed throughout 
counselors, job developers, and management. I believe that the 
metropolitan environment of this office elicits a mora realistic 
approach to social change on the part of this staff. A computerized 
job matching system perhaps seemed too unrealistic, and time required 
to process an applicant (SICL acbiilnl strati on through placement) proved 
too consuming and Involved to undertake without considerably more 
evidence that their efforts would achieve worthwhile results. 




Organization and sound, basic management seem to permeate this office. 
Well-established Intake, process and follow-up procedures enabled this 
office to undertake requirements for the Cleff evaluation with only 
moderate deviation frcm routine responsibilities. CJMS technician 
worked relatively well through SICL Introduction and administration. 
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EXHIBIT 4-2 (continued) 



12 July 1973 
Page 4 



mUKContl nued) 

In all, IHH^^B responded most positively to the system, 
realizing early Its potential. Staff counselors Initiated the 
system's use as a selection tool for acceptance or rejection to 
training classes. Staff members also conducted client analysis, 
Interpreting computer p rintouts and coun seling clients with Cleff 
Information as a base. ^m^^HHH Sr. Counselor at this 
office, has a firmly es tabltshed po sture as to his response to 
the system. By request, ■■■■ has received additional guidance 
and Instruction In the periphery of the Cleff System and I can feel 
confident he can effectively utilize the system In those areas 
normally e ntertained by comnunlty service agencies. Additionally, 
■M^H has expressed his awareness that time was their Qnmy\ 
that his office, too, had work loads far too heavy for effective 
involvement In the Cleff project. Despite these handicaps, the 
San Jose office processed a significant number of SICL's. Cooperation 
dnd participation was as high as could be expected under the clrcL.^- 
stances. 



JOCL introduction and participation had some Initial schedule conflicts, 
but job developers made necessary adjustments, opening avenues to good, 
solid JOCL instruction. Enthuslasn and Interest flowed smoothly from 
counselor phase to job developer phase. Again, normal SER assignments 
and responsibilities severed additlonJil Involvement of job developers 
In JOCL acquisition, minimizing their roles to that of occasionally 
establishing Initial contacts and supplementing Information available 
to Ultrasystems personnel about those contacts. 

Business and Industry generally responded well to the research project. 
The acquisition of JOCL appointments was largely routine. There were, 
naturally exceptions. All things considered, acquisition of JOCL's 
at this location should continue at an acceptable level. 

SUMMARY 



Wherein one of our objectives was to provide the SER Project Offices 
an opportunity to work with the CJMS to assess its effectiveness and 
usefulness In their operations, I don't believe the offices allotted 
enough time for CJKS Involvement to make such an assessment. Had there 
been a better understanding Initially between Ultrasystens and the 
SER Project Offices to outline and detail the Interrelation of both 
entitles as related to this evaluation, better time constraints would 
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EXHIBIT 4-2 (continued) 



12 July 1973 
Page 5 



have been established, A large portion of the confusion is attributed 
to the local office directors being unaware of the degree of involve- 
ment required on the part of their staff members. The San Jose 
office personnel will render the most significant data relative to 
reactions and opinions of the CJMS, Additionally, the depth of their 
involvement will maximize your dependent variables. In commenting on 
the advantages of using CJMS, I see them as being no different than 
those advantages experienced in Neterk, The system, when properly 
implemented, will ideally confinn savings in cost, time, and improve- 
ment on job matches, I don't think the system is as "tough" as it 
may have to be to survive in a realistic atmosphere. To be effective, 
the system would either have to undergo stringent ct'tange to "toughen" 
itself for handling in a less sensitive atmosphere, or, if allowed to 
go unchanged, be implemented under close scrutiny and supervision of 
qualified personnel. The current system with improper implementation 
is, in my estimation, too sensitive to survive long enough to provide 
valid data upon analysis. The SER organization is well qualified in 
tenns of types of ot^anizations, to benefit from CJHS usage. Used as 
selection criteria for training, used as a counseling tool, used as a 
job matching mechanism, SER could save on cost and time in those 
areas, and improve on the results of all three. To be aware of this 
potential saving, however, the organization will have to set aside 
time to get involved and properly evaluate the system. 
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EXHIBIT 4-3 



In administering the SIC(.» niy greatest concern has been the experience 
portion of the questionnaire. While a very few people caught on quickly 
and had little trouble relating the activities to their personal lives, 
the majority found this confusing, some to a great extent. Frequently 
an Individual would be able to relate the activities to his work 
experience only with constant Interpretation on my part. If the SICL 
was being administered to a large group, this was Impossible. 
Therefore, the validity of the experience questionnaire was sometimes 
doubtful. 

In general, the preference section was much more easily understood by 
most people. One thing I noticed, however, was that Items representing 
the same category or different pages sometimes seemed very dissimilar 
and therefore drew opposing reactions. Perhaps this was Intentional, In 
an effort to represent the wide variations within a category, but I 
wondered If varying reactions might not tend to cancel each other out at 
times and then register as neutral. 

The only other coiiment I have regarding the SICL itself Is regarding the 
Spanish translations, I was told there have been several revisions of 
it and that Spanish-speaking people In the east helped work on It, but 
according to the Mexican trainees who took It here, many of the items 
were poor translations, I think this Is due to Idiomatic expressions 
included, which vary from culture to culture (Puerto Rican, Mexican, 
South American, etc). In certain Instances students had no Idea what 
a particular phrase meant. 

Along these lines, my Spanish Is not fluent, and although I had Spanish- 
speaking teachers helping me there might have been more problems In 
comprehension than I was aware of. 
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EXHIBIT 4-3 (continued) 



In regard to the cooperating staff at SER, although everyone was 
pleasant, I'm nto sure how seriously they took the SICL. Results 
were sometimes received with great Interest, end at other times with 
very little. I also wondered how well informed the various staff 
members were regarding the whole project, as X was frequently asked 
questions concerning Its purpose, use of results, what various staff 
members were required to do, etc. These were really out of my 
territory an(? authority and I regularly suggested they contact Ron or 
Arnold, or wait until their next visit. 

So far I have specific information about job placement for only 19 
trainees. Of course, a large number of those who took the SICL are 
still In English classes. As more results come in, I will send them 
on. 

I hope these coinnents help. If I can be more specific, let me know. 
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5.0 



ANALYSIS OF THE DATA OBTAINED FROM THE SER/CLEFF 
EXPERIMENT 



This section presents the data obtained froin the SER/Cleff experi- 
ment and the analyses undertaken. The section has been divided 
into four subsections at follows: 

• Client Demographic Data {Section 5.1) 

This subsection will present the client demographic data 
for the 142 clients included in the retention followup 
data base. This data will be compared to data showing 
the demographics of SER clients. 

• Job Description Data {Section 5.2) 

This subsection will present data describing the jobs 
that were profiled. 

• Client and Job Cleff Indice Data and Analysis {Section 5.3) 

This subsection will present data showing the distribution 
of client and job scores fOr the 142 clients and 110 job 
profiles included in the analysis. The statistical analyses 
completed using this data will be presented. 

• Retention Analysis {Section 5.4) 

This subsection will present the retention data obtained 
and the analyses performed relating the Cleff system 
scores to the job retention data. 

There is no data presented about all the clients who took SICLs, 
aboi't their SICL scores* and bhe services provided by SER. In 
other words, all the data and analyses presented in this section 
relate to the 142 clients whose retention was tracked and the 110 
job profiles which were obtained. 

Table 5-1 presents somer surtmairy data to give the reader a feel for 
the data base before the detailed data and analyses are presented. 
The data presented in this table shows the following: 

1) A total of 571 people took one or both SICLs. Of this. 
142 clients were Included in the experimental data base. 
As of October 31, 1973, 303 of these 571 people had been 
placed. This is the last time at which Ultrasystems re- 
viewed the SER case files for placement data. Tne outcomes 
of the other people who took the SICLs is not known. 
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TABLE 5-1 

SUKHARY DATA REGARDING THE SER/aEFF EXPERIMENT 



SUMMARY 
DATA 




SER AGEHCY CODE UUMnER 






r X of J 
Totar 


1 


2 


3 


4 


Total 


1. Total People 
Profiled 


89 


193 


139 


150 


571 




2. Total Included 
In the Retentio 
Data Base 


1 

35 


. 42 


28 


37 


142 


[24. 9X: 


3. Total Hired Per 
SER Records as 
of Oct, 31 ,1973 


75 


76 


71 


81 


303 


[53, u; 


4. Number of Job 
Profiles Used 
In Analyses 


30 


29 


24 


27 


no , 




5. Ntmber of 
Individual 
Enployers at 
mMch Profiles 
Were Obtained 


27 


' 28 


20 


25 


100 




6. CuMulatlve 
Number of 
People In Data 
Btse Tennlnatecl 
as of Retention 
Followup Time: 














Oecendier 30.73 
Febr. 28 » 74 
Hay 31 ► 74 
Decetnber 10*74 


11 
12 
14 
17 


31 
34 

35 
36 


12 

15 
17 
18 


13 
19 
20 
23 


67 
80 
S6 
94 


(47.21 
(56.3) 
(60.6) 
(66.2) 


7. Number & Percer 
of People In 
Data Base Who 
Completed 
Both SICLs 


t 

27 07t) 


34 


21 (750 


28 (76X) 


110 


(775t) 



Percentages In square brackets use the total of 571 clients (who 
took «t least one Sia) as the denominator. Percentages In parentheses 
uje the 142 clients Included In the experimental sample as the denominator. 
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TA81E 5-1 (continued) 
SUHWRY DATA tttGARDING THE SER/CLEFF EXPEttlHENT 



SUMMARY 
fiATA 




SER AGEHCY COnF HUHBER 






% of 
Total 


1 


2 


3 


4 


Total 


8. Mean Cleff 
Indices: 












■ 


CAI 
COI 
JHI 
JDI 


49.6 
939.3 

33.2 
1999.3 


38.3 
1211,7 

25.6 
2242. < 


22.7 
1468.0 

27.0 
1856.4 


34.6 
1525.8 

30.9 
2439, & 


37.1 
1273.3 

29,1 
2157,8 , 




9, Humber of Client 
Uho Terminated 
by Calendar Days 
Worked 














«30 

31 - 60 
61 - 90 
91 - 120 
121- 199 
200- 584(longest 
lact 
termination) 

Total Terminations 


4 

2 
3 
0 
2 

17 (48.6) 


16 
4 

2 
2 
9 
_3 

36 (85.7) 


3 
2 
1 
3 
5 
-1 

18 (64.3) 


5 
2 
3 
2 
4 
J.- 

23 (62.2) 


28 
10 
9 
7 

20 
_20 

94 


(19.7) 
( 7.0) 

( 6.3; 

( 4.9} 
(14,1) 
(14.1) 

(66.2) 
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2) Of the 142 people in the experimental data base* 110 took 
both SICL booklets (77SS). 

3) As of the last retention foUowup (Decerrtber 10, 1974). 94 
of the 142 people had tenninated {66,2SS), Twenty-eight 
terminated after working 30 or less (calendar) days 
(approximately 30SS of all the terminations), and 47 people 
terminated after working 90 or less (calendar) days (50% 
of all terminations). 

Therefore, as of December 10. 1974. 48 people were still 
working (33,8SS). 

4) The mean client adjustment and difference Indices (for the 
110 people who took both SICLs) were +37,1 and +1273,3. 
respectively, 

5) The mean job match and difference InxJices for the experi- 
mental sample were +29,1 and +2157,8. respectively. 

(NOTE: The values for the client adjustment and job 
match indices (which are correlation coefficients) are 
Sfflnetimes given as 100 times the correlation coefficient 
value. Thus, a client adjustment index may be given as 
+,5 or +50,) 
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TABIE 5-2 

SUWMRY OF SER EXPERIHENTAL GROUP CIMRACTEfllSTICS 



1 



SER AGEttfY CODE umm 



Total 



% of 
Total 



Total In Sample 
(Percent of 

Sample) 



35 
(24.6) 



42 
(29.6) 



28 
(19.7) 



37 
(26.1) 



142 



(100) 



Character-! Sties 

Sex: Hale 
Female 

m 



13 (37.1) 
22 (62.8) 
0 



12 (28.6) 
30 (71.4) 
0 



8 (36.3) 
14 (63. S) 
6 



19 (51.4) 
18 (48.6) 
0 



52 
84 
6 



(38.2) 
(61.8) 
[ 4.2] 



Ethnfc: 
Mexican- Americai 
Cuban 

Puerto (Ifcan 
Other Spanish 
Afro American 
Anglo American 
American Indian 
Other 

INA 



22 
0 
0 

13 
0 
0 
0 
0 



(62.8) 



(37.1) 



32 
0 
2 ( 

n 

2 ( 



(80.0) 



5.0 
5.0 
2.5 
5.0 



20 (90.9) 
0 



1 ( 2.5 
0 



(4.5) 



(4.5) 



28 (75.7) 
0 . 
( 2.7) 
8.1 
( 8.1) 
( 2.7) 



( 2.7) 



102 
0 

3 

19 

4 
3 
2 
1 



8 



(76.1 : 
p , 

( 2.2; 

(14.2 
3.0 
2.2 
1.5 
O.7J 



[ 5.6] 



Age: 

18 & Under 
19-21 
22-32 
33*44 
45 i Over 

Average 



6 (17.1) 
13 (37.1) 
12 (34.3) 
3 8.6) 
1 i 2.8) 



8 
11 

13 
5 
3 



(20.0) 
27.5] 
32.5 
(12.5 
7.5 



24.11 



2 

25.7 



5 
6 
11 
0 
0 



21 .8 



10 

9 

15 
3 
0 



23,1 



29 
39 
51 
11 
4 



8 

23.9 



(21.6/ 
29.1, 
38.0 

8.2; 

3.0 



r 5.6} 



Educatior (State)) 

4th or Less 0 

5th-7th 0 

eth 3 

9th-llth 19 

12th * Over 12 

m 1 

Average 10.9/34 



( B.8 
(55.9 
(35.3 



2 
3 
2 
21 

11 

3 



{ 7.7) 

5.1 
(53.8) 
(28.2) 



10.2/39 



0 
2 
2 

12 

5 
7 

10.4/21 



i57!ii 

(23.8) 



( 



0 
2 
3 
18 
10 
4 

11 .0/33 



6.1 

5.1) 
54,5) 
30,3) 



2 
7 
10 
70 
38 
15 

tt0.4/l2l1 



( 1.6) 
5,5) 

l55!li 
(29,9) 
[10,6] 
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TABLE 5-2 (continued) 
SUmV OF $£R EXPERIHEMTAL GROUP CHARACTERISTICS 



SER AGEHCY COOE HUHBER 





1 


2 


3 


4 


Total 


t * of 
Total 


Total in Sample 


35 


42 


28 


37 


142 


(100) 


Characteristics 














Education 
(Functional): 
6th or Less 
7th-9th 
10th & Over 
INA 


INA ■ 
INA . 
INA 

35 


7 

16 
0 
19 


INA 
INA 
INA 

28 


INA 
INA 
INA 

37 


119 


[83.83 


Average 


INA 


7.23/23 


INA 


■INA 






Dlsadvantagecl 
INA 


9C /71 A\ 

10 


to tQt\ ti\ 

4 


9 


ou 1 » t ^ 
7 


1 t£ 

30 




Public Assistanci 
Recipient (Yes) 
(No) 
INA 


4 

25 
6 


7 

33 . 
2 


5 

17 
6 


9' 
25 
3 


25 
100 
17 


(17.6) 
02. 0] 


Heads of Family 
or Household 
■ Yes 

m 

INA 


20 
14 
1 


26 
14 
2 


11 
11 
6 


25 
12 
0 


82 
51 
9 


(61.6) 
(38.3) 
[6.33 


Honolinguals 
Yes 

rio 

INA 


12 
14 
9 


5 

35 
2 


6 
14 
8 


11 
19 
7 


34 
82 
26 


(29.3) 
(70.7) 
[18,3] 


Migrant Yes 
No 
INA 


2 
4 

29 


2 
37 
3 


6 
14 

8 


3 
24 
10 


13 
75 
50 


( 9.2) 
[35.2] 
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TABIE 5-2 (cont-tnued) 
SUHHARY OF SER |XPER1MENTAI GROUP CHARACTERISTICS 



SER AfiEHCY CODE HUMBER 





1 


2 


3 


4 


Total 


% of 
Total 


Characteristics 














Citizenship or 














Permanent 














Resident: 














Citizen ' 


6 


30 


9 


13 


S8 


(43,31 




Pern. Resident 


8 


5 


6 


6 


2S 


(i8.e! 




Yes 


10 


4 


5 


13 


32 


(23.9: 




Resident 


10 


1 


1 


3 


15 






No 


1 


0 


1 


2 


4 






IHA 


0 


2 


6 


0 


8 


t 5-6] 


1 


Employment 














Status 














Employed 


3 


1 


3 


4 


11 


( 8.7; 




Underemployed 


2 


0 


4 


1 


6 


( 4-7 




Unemployed 


27 


39. 


14 


29 


109 


(85,8 




INA 


3 


2 


7 


3 


15 

1 


[10,6. 
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TABIE 5-3 

SUmARY OF SER £NROLLEE CHARACTERISTICS 

.Cumulative Statistics (Time Period 9-18-72 

to 5-31-73: 8.5 months) 



SER AGENCY CODE MUHBER 





1 


2 


3 


4 


Total 


t % Of 
Total 


Total PnrAlltnAnt 
fP€irr€nt of 
4 kQm\rV Total 


288 
(22.0) 


333 
(25.4) 


298 
(22.7) 


392 
(29.9) 


1311 


(100) 


fbiracterl sties 
S#x* Hale 

• fin 1 c 

Female 


133 1 
155 1 


[46.2) 
[53.8) 


103 1 
230 ' 


;30,9) 

1 69.1) 


191 1 
107 1 


[64.1) 
[35.9) 


176 (44.9) 
216 (55.1) 


603 
708 


( 46) 
( 54) 


Ethnic: 




























Mmcl ^a n**Aiwv* 


176 1 


61.1) 


239 1 


71.8) 


270 ( 


90.6; 




295 1 


[75.2] 




9S0 


1 


74.8) 




9 1 


3.1) 


17 i 


! 5.1) 


1 1 


0.3, 




2 


o.s! 




29 




2.2} 


Pii#l^o Riran 


14 1 


4.9) 


6 


[ 1.8] 




1 1 


0.3, 




5 


1.3; 




26 




2.0 


nthsr Soanlsh 


85 1 


.29.5) 


17 


5.1 




8 1 


2.7 




16 J 


4.1 




126 




9.6) 


Afro- American 


0 




16 


4.8! 




3 1 


; 1.0 




17 


4.3 




36 




2.7) 


Anglo-American 


.3 ( 1.0) 


29 


12.7; 




8 1 


, 2.7: 




41 1 


10.4, 




81 




6.2} 


American Indlar 


0 




3 






2 


: 0.7; 




6 1 


1.5 




11 




o.e) 


Other 


1 ( 0.3) 


; 6 1 


: i-s! 




5 1 


: 1.7] 




10 1 


2.6 




22 




1.7) 


Age* 




























[13.21 


18 and tinder 


24 1 


; 8.3) 


60 1 


18.0] 




38 1 


12.8, 




51 ( 


13.0 




173 




19-21 


81 1 


28.1) 


88 1 


:26.4 




82 1 


27,5. 




126 1 


32.1; 




377 




[28.8 


22-32 


136 I 


47.2) 


114 1 


134.2 




131 1 


44.0 




167 1 


42.6 




548 




41.81 


33-44 


37 1 


12.8) 


59 1 


17.7 




39 1 


13.1 




42 


10.7 




177 




13.51 


45 and Over 


10 


[ 3.5) 


12 1 


; 3.6] 




8 \ 


: 2.7' 




6 1 


: 1.5 




36 




; 2.7) 


Education 




























(Sttted): 




























4th or Less 


6 < 




12 


[ 3.6 




11 


, 3.7) 


4 


1.0 




33 


( 2.S) 


5th-7th 


19 


! I'i] 


26 


: 7.8 


J 


61 


:20.5 




30 


7.6, 




136 


(10.4) 


8th 


16 


: 5.6) 


21 1 


6.3] 




15 1 


: 5.0 




20 


: 5.1 




72 


S.S 


9th-lUh . 


127 \ 


44.11 


134 1 


,40.2 




121 1 


40.6 




188 


48.0 




570 


(43.5 


12th and Over 


120 


[41.7) 


140 1 


;42.ol 




90 1 


30.2 




150 


38.3] 




500 


(38.1) 


Education • 
(Functional): 
ftth or Less 
/th-9th 
10th and Over 


156 (54.2) 
132 (45.8) 


165 (49.5 
143 (42.9 
25 ( 7.5 


J 
1 


INA 
INA 

IKA • 


INA 
INA 
INA 


321 

275 
25 




51.7 
[44.3) 
( 4.0) 
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TABLE 5-3 (continued) 

SUHHMIY OF SER ENROLLEE CHARACTERISTICS 

Cumulative Statistics (Time Period 9-18-72 

to 5-31-73: 8,5 months) 

SEB AGEMCY CODE HUHBER 



Total 



% Of 

Total 



Characteristics 

Disadvantaged ^ 
Heads of Family 
or Household 
Public Assist- 
ance [tec i pi en t: 
Monolingual s 
Vietnam Era Vet< 
Migrant 



288 (100.0) 
114 (39,6) 



10 
74 

6 ( 2 
0 



( 3,5) 
(25.7 



i 



299 

227 

99 
65 
19 
17 



(89.8) 

(68.2) 

(29,7) 
(19,5 
( 5.7 
( 5.1} 



296 (99.3) 

234 (78.5) 

74 (24.8) 
114 (38.2) 
16 ( 5.4) 
93 (31.2) 



384 (98,0) 

272 (69,4) 

97 (24,7) 
71 (18.1) 
28 (.7.1) 
0 



1267 

M7 

280 
324 
69 
110 



(96.6) 

(64,6) 

(21.4) 
(24.7) 

5 

8 
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Each of the two tables shows the demographic distribution 
for each of the four SER agencies separately and then in- 
toto across all four. The percentages in ttie columns 
under each agency are the percent of that agency's experi- 
mental group that were characterized by the particular 
value of the demographic characteristic. All percentages 
shown in the curved parentheses are adjusted percentages; 
i.e., they are based not on the total agency sample but 
on the nun^er of people for whom we have a value for that 
characteristic. In other words, the INA counts are,ex- 
cluded prior to the calculation of the percentages. 
In the % of Total column, the percentages shown in the 
square brackets for the total irJA counts are the percent 
of the total experimental group for which the particular 
characteristic's value is not known, i.e., the percent of 
INA's. 



Host of the demographic charactenstic data items are 
self-evident, A few deserve to be defined (the defini- 
tions are taken from the SER MIS manual). 



HEAD OF FAMILY 
OR HOUSEHOLD 



Check "Yes" if the applicant normally, 
when employed, is primarily responsible 
for the support of one or more members 
of his family, or if the person is 
regarded as the head of household by 
its members. Otherwise, check "No." 



PUBLIC ASSISTANCE 
RECIPIENT OR 
UNEMPLOYMENT 
INSURANCE CLAIMANT 



Check "Yes" if at the time of the inter- 
view the applicant was receiving financial 
aid under the following categories: 

—Old Age Assistance (Title I) 

—Aid to Families with Dependent Children 

(Title IV) 
—Aid to the Blind (Title X) 
-Aid to the Permanently Disabled (Title XIV) 
—Aid to the Aged, Blind, or Disabled 

(Title XVI) 



There are three exceptions to this. These are the total percent- 
ages given for disadvantaged, public assistance recipient, and 
migrant, where the percent of Yes responses is given in relation 
to the total experimental sample, i.e., ths 142 people. 
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—Any other Federal, State, or local 
public assistance program. (Do not 
include surplus food, food stamps, 
assistance from private, voluntary agency, 
pensions, or old-age and survivor's in- 
surance benefits). 

"If the applicant is uncertain about the 
source of any aid he is receiving, the 
interviewer should check with the counselor 
who is making the determination of the 
applicant's eligibility for a training 
allowance. 

A youth whose parent{s) is receiving AFDC, 
or other State or local aid, is to be checked 
"Yes" if his needs are considered in comput- 
ing a public assistance grant to his family. 

Check "Yes" if the applicant is claiming 
Unemployment Insurance benefits at the time 
of the interview. 

If the answer is "Yes," then specify which 
one, otherwise check "No." 

MONOLINGUAL Check "Yes" if the applicant speaks Spanish 

but has a limited English-speaking ability 
which severely limits his opportunities for 
meaningful employment. English-speaking 
ability refers to the ability to converse in 
English and to understand spoken English; a 
person should not be included in this cate- 
gory because of ignorance of proper graiimar. 

MIGRANT Check "Yes" if the applicant is a migrant 

farmworker. A migrant farmworker is a per- 
son who left his home temporarily overnight 
to do farm work within the same State or who 
had no usual place of residence and did farm 
wage work in two or more counties during the 
year. 

FViLTlYMENT STATUS Check only the most appropriate item to 

indicate the applicant's present status. 

Emploved— Check employed if the applicant 
did work as a paid employee, in his own 
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business, or on a fann, or worked at least 
15 hours during the past week as an unpaid 
worker in a family enterprise. Check employed 
also if the applicant was absent from work 
because of illness, vacation, bad weather, 
labor-management dispute, or other personal 
reason. Do not include those whose activity 
consisted of volunteer work for religious or 
other charitable organizations. 

Underempl oyed— Check if the applicant is 
working less than full time in his occupa- 
tion for economic reasons such as slack work, 
material shortage, plant repairs* etc., or 
has received notice that he will be employed 
less than full time for economic reasons, 
or he is working below his skill capacity, 
or has received notice that he will be 
unemployed because his skills are becoming 
obsol ete. 

Unemployed— Check if the applicant is a 
civilian who has no employment, who is 
available for work, and who has engaged In 
job seeking activity such as: registering 
at a public or private employment office, 
meeting with prospective employers, check- 
ing with friends or relatives, placing or 
answering advertisements, or being on a union 
or other register. Also, include an eppllcant 
waiting to be called back to a job from which 
he had been laid off, or an applicant waiting 
to report to a job scheduled to begin In the 
following 30 days. Students who are seeking 
vacation jobs or work outside of school hours 
are also considered unemployed. Persons regis- 
tering through any manpower program will be 
considered unemployed, and though the applica- 
tion may be made for training rather than an 
inmedlate job. 

DISADVANTAGED A disadvantaged individual, for Manpower 

Program purposes, is a poor person who does 
not have suitable employment and who is 
either 1) a school dropout, 2) a member of 
a minority, 3) under 22 years of age, 4) 
45 years of age or over, or 5) handicapped. 
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The functional education level is obtained from the tests administered 
to prospective enrollees during Assessment and Orientation, 

If one refers back to the SER form S-1 shown in Section 4» one sees 
that the entries Head of Household* Public Assistance Recipient* 
Monolingual* and Migrant are Yes/No entries. When no value was checked, 
we classified the response as INA. The Monolingual distributions shown 
In Table 5-2 included special cases where the entry given on the S-1 
was neither Yes nor No. These special entries were "marg»" which 
we took to mean marginal, and coded as "Yes" and "Bi»" which we took 
to mean "Bilingual" and coded as "No." It is important to note that 
the definition for Monolingual given above refers to speaking ability 
and not to reading or writing ability. 

The data shown for Head of Household was taken on face value based on 
the Yes/No entry for this item. It is Ultrasystems' opinion that this 
entry is not clearly understood by most of the applicants. The S-1 
form contains other entries giving family size» number of dependents, 
and marital status. While these are not given In Table 5-1 » our 
examination reveals many inconsistencies between these entries and 
the answer given regarding Head of Household, 

The data for the entry on the S-1 form for Citizen or Permanent 
Resident is structured as a Yes/No answer, where if the answer is 
Yes then the appropriate answer is to specified. The data shown 
in Table 5-2 is as it was recorded on the S-1 forms, Thus» the answer 
Yes appears with no specification. 

Comparing the data In the two tables shows that in general the experi- 
mental sample is comparable to the cumulative enrollee characteris- 
tics. There are several differences, as follows: 

1) The experimental sample contains a larger percentage of 
women than comprise the cumulative SER enrollee population. 



2) The experimental group has a larger distribution in the 
18 and Under and 19-21 year-old age groups and smaller per- 
centages in the older grouos. Whereas no average a9e is 
available from the SER Cumulative Enrollment data, \t appears 
that the experimental group is younger than the cunulative 
enrollee population. 
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3) The experimental groyp has a smaller percentage of its 
population in the two lowest educational classes (edu- 
cation as stated) and in the highest. The percent of 
the experimental sample with at least some high school 
is slightly greater than the cumulative enroUee popu- 
lation {B5% as compared to 81.6%). 

Table 5-4 shows the Stated Education distribution as a 
function of the country of the school last attended. The 
adjusted percent distribution shows that for the 122 
people whose education level and country is known 68% 
last attended school in the United States; 23% in Mexico; 
and the ranaining 9% in Central and South America. Be- 
cause of the large percentage of people who were educated 
outside the United States, care must be taken in interpret- 
ing this data item, especially since the corresponding 
distribution for the cumulative enrollee population is not 
known. 

4) The remaining comparable characteristics are in reason- 
ably close agreement with the exception of the percent of 
the two populations classified as disadvantaged. Ultra- 
systems believes the apparently lower percentage of dis- 
advantaged people in the experimental sample is due to the 
lack of complete information in our data base. 

Table 5-5 presents data showing the number of new applicants and 
new enroUees' for the three months of March, April, hay 1973 and 
over the 8-1/2 month period ending Kay 31, 1973. The bulk of the 
SICLs administered as part of this project were done in this three 
month time period, while the majority of those done in June were 
completed by enrollees. The data in this table indicates that 
approximately 2Q% of the total people who could have potentially 
taken the SICLs actually did so. 
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TA8LE 5-4 



DISTRIBUTION OF HIGHEST GRADE COMPLETED (EDUCATION), 
AS STATED, AS A FOJNCTION OF THE COUNTRY OF THE LAST 
SCHOOL ATTENDED 



Country of Last School Attended 
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Educati On SKR'^'r'^'o Sw< 

As Stated :s:eiuoocjcqiuo.*-i 



4th or Under 


1 


1 


0 


0 


0 


0 


0 


0 


0 


0 


5th- 7th 


0 


5 


0 


0 


0 


0 


0 


0 


0 


2 


8th 


4 


4 


0 


0 


1 


0 


0 


1 


0 


0 


9th- nth 


55 


11 


1 


1 


0 


0 


0 


0 


1 


1 


12th & Over 


23 


7 


3 


0 


1 


1 


1 


0 


0 


2 


I»A 


2 


2 


0 


0 


0 


0 


0 


1 


0 


10 



Total 


85 


30 


4 


1 


2 


1 


1 


2 


1 


15 


Total (^own 






















Grade Level 


83 


28 


4 


1 


2 


1 


1 


1 


1 


5 


Percent of 






















Known Total 


(68.0) (23,0) 


(3.3) 


(0.8) 


(1.6) 


(0.8) 


(0.8) 


(0.8) 


(0.8) 


[3.9] 
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TABLE 5-5 

SUMMARY SER OFFICE 
APPLICANT AND ENROLLMENT VOLUMES 



Applicant & 



Enrol lee Volumes 
as ueTinea 




1 7 
c 


•1 




Tota I 


Total Applicants 
March 1973 
Apnl 1973 

Mai/ 1 a71 

nay ly/o 


206 
164 


169 
138 

117 
1 I / 


305 
225 

^ /3 


165 
138 


845 
665 

/JO 


Total 


525 


424 


805 


492 


2246 


3 Month Average 


175 


141 


268 


164 


749 


Cum, Total thru 
May 31, 1973 


1572 


1675 


2192 


1673 


7112 


Months Average 


185 


197 


258 


197 


837 


Eitrollees 
March 
April 
May 


63 
23 
20 


21 
26 
24 


4 

45 
21 


37 
32 
74 


125 
126 
139 


Total 


106 


71 


70 


143 


390 


3 Month Average 


35 


24 


23 


48 


130 


Cumulative 
thru 5-31-73 


288 


333 


298 


392 


1311 


Months Average 


34 


39 


35 


46 


154 


Enrollment at 
Start of March 
1973 


77 


98 


94 


120 


389 


Enrollment at 
End of May 1973 


64 


58 


106 


104 


342 
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5,2 



JOB AND JOB PROFILE CHARACTERISTICS 



A total of no individual job profiles were administered based on 
the jobs obtained by the 142 people who comprise the experimental 
sample. The difference in the two numbers is due to the fact that 
in 21 instances more than one person was placed into the same type 
of job at the same employer. The maximum nunter of people placed in 
a single type of job at a given employer was 5, The 110 job profiles 
were obtained at 100 separate employers. Table 5-6 presents job 
and job profile characteristic data for these 110 job profiles. 
The table presents the following data: 



• The JOCL number assigned by Ultrasystems. The first 
digit identifies the agency who referred the people 
to this job. 

• The job title as given by the employer. 

• The type of company as defined by Ultrasystems based 

on the information obtained during the JOCL administra- 
tion. Manufacturing companies are identified by the 
products produced at the physical plant v^ere the 
person was working. Non-manufacturing companies are 
identified by the services provided. 

• DOT code assigned by Ultrasystems based on the job 
description obtained from the employer. 

• The wages paid the person (or persons) placed at the 
time of the placement. All wages have been converted 
to dollars per hour. 

• The number of the job profile cluster that best "matches 
the JOCL profile obtained. Each JOCL was correlated 
with the 19 job cluster profiles contained in the CJMS 
Cluster Register, The "best matched" cluster is the 
one which has the highest correlation with the profile 
obtained. The correlation coefficient is also shown. 

• The number of experimental group clients placed into 
the job profiled. A number is given only when there 
is more than one person placed into a particular job 
at a particular company; i,e., all the jobs without a 
number in this column had only one person placed. 
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TABLE 5-6. CHARACTERISTICS OF JOBS AND JOB PROFILES 



Ctoploytr Job Titlft 



t 

to 

CO ^ 
CO 



1001 K«tl Clert/Typtit 

1002 Ciihter 

1003 HTST Operctor 

1004 ttarehouianan 

1005 Floor lfeiieng«r 

1006 Accounting Clerk 

1007 File ClerX 

1008 Gener«1 Clerk 

1009 General Clerk 

1010 Food Servtce UoHcer 

1011 Secretcry 

1012 Ftle Clerk 

1013 . Parking Attendant 

1014 Kachtne Operator 

1016 Intennedtate Cirk./Typtit 

1017 Machine Adjuster 

1018 Sales Clerk 

1019 Janitor 
10Z1 Ftle Clerk 
10e3 Bank Teller 

1024 Face Sheet Clerfc 

1025 Sub<^aii«nb1er 

1026 Fraineman 

1027 Geiieral Clerk 

1028 Saleinan 

1029 Stock Clerk/Delivery 

1030 Data Entry Operator 

1031 KeyPtmch operator 

1032 Clerfc l^plit II 

1033 Teller 



Typt of CMpti^y 



Auto Parti Kf9, 

Grocery Stor« 

Teaching Manuals Publisher 

Ut9 Ua rehouse 

Insurance 

Insurance 

Insurance 

Telephone Co. 

lelephone Co. 

Atrline 

Health Center 

Insurance Group 

Parking lot 

lock Assembly Mfg. 

Hospital 

Envelope Mf9. 

Hen's Oepartioent Store 

Auto Part ftebuUdlng 

Insurance 

84nk 

Social Servtce Dept. 
Resistor Assembly 
Telephone Co. 
Credit Verifying servtce 
Electrical Sales Supply 
Onig Store 
School olstrtct 
Insurance 

Dept. of Motor Vehicles 
8ank 



DOT 
Code 



237.368 
211.368 
203.582 
922.687 
230.878 
210.368 
206.^86 
209.388 
209.333 
315.381 
201-368 
206.383 
915.878 
649.885 
209.388 
641.780 
290.468 
302.884 
206.388 
212.363 
209.388 
729.884 
222.834 
209.388 
278. 3S8 
223.387 
231.582 
231.582 
209.388 
212. 3$8 



Wtgti 

Ooll art/Hour 


Cluster 
. tMer 


viuiter 
natch 


3.06 
2.67 


12 


84 


6 


63 


2.88 


9 


80 


2.50 


11 


74 


2.25 


17 


80 


2.57 


12 


76 


2.23 


11 


87 


2.97 


12 


82 


2.97 


12 


83 


4.00 


17 


83 


2.89 


11 


82 


2.50 


11 


76 


2.25 


3 


80 


2.50 


14 


84 


3.18 


12 


84 


3.19 


5 


90 


2.25 


6 


67 


2.50 


18 


82 


2.30 


11 


90 


Ml 


12 


87 


3.18 


12 


86 


2.70 


5 


81 


4.67 


7 


63 


2.25 


1 


88 


2.50 


6 


74 


2.50 


8 


84 


3.18 


12 


82 


3.18 


12 


80 


2.79 


12 


89 


2.45 


6 


93 



of 

Flactntntf 



TABLE 5-6 (continued) 



i. to 





Effplcytr Job Titit 


Typt of Conptny 


DOT 
Code 


Magts 

Oollars/Hour 


Cluster 
Htflber 


Cluster 

naccn 

inocx 


rluff6er of * 
PlaceMtnts 






OliPosal Service 


619.130 


3*75 


2 


78 




2C02 


Cleri; Typfit 


Cngr^/Keavy Construction 


209*388 


2*54 


12 


93 




2003 


Utility Kan 


£hgr./Kf9. Flight Controls 


3SK887 


2*80 


18 


96 




zoos 


Clerk Typist 


Yacht Kfg* 


209*383 


2*50 


12 


81 




2006 


Hech* Assembler 


Clectronfcs Hf9* 


726*781 


2*00 


12 


74 




2007 


t/H Mseotbler 


Computer KfOi. 


726*781 


2.35 


9 


85 


3 


2006 


Repaf rman 


Vacuum Service 


723.CB4 


3.60 


5 


83 




2009 


t/H Aiseo^ler 


Electrical Connectors Hf^* 


726,781 


2.1S 


12 


71 


3 


201 G 


Seamitreii 


Garment Kfg«. 


785*381 


2.00 


12 


82 




2011 


C/H Aiienbler 


Sv«1tch Hf9. 


726*781 


2.25 


12 


88 


5 


2012 


t/n Assembler 


Tape Recorder Kf9«. 


726*781 


2.50 


12 


86 


2 


2014 


01 rectory Asit* Op^ator 


Telephone Co, 


23S.862 


2 80 


1 


89 




2015 . 


Furniture Aiiewbler 


Furniture 'tf9* 


763.384 


2ls0 


IS 


83 




2016 


Wilder Trainee 


Specialty Metal CqufP. Kfg. 


812.884 


2.00 


18 


83 




2019 


COMnunlty Afde I 


Welfare Office 


194.168 


2.48 


3 


51 




2021 


Receptionist 


Guitar Hfq* 


237. 3$e 


2*40 


1 


62 




2022 


Finisher 


Ffshing Rod Hfq* 


739*884 




12 


82 




2024 


Asieoibler 


eatterx Mf9. 


726*781 


2,2S 


16 


83 




202S 


Inspector 


Oattery Kfg« 


727*687 


2*50 


14 


77 




2026 


CleCi. Asietnbler 


Printed Circuit Ooard Hfg. 


726.781 


2*20 


12 


88 




2027 


Cleric Typllt 


Gas Meter Kfg. 


209*388 


2*70 


12 


78 


2 


202S 


Gardener 


Land Developers 


407.884 


3.00 


15 


84 




2029 


Clerk 


Insurance 


209*338 


2.45 


1 


91 




2030 


Credit Verifier 


8ank 


191*268 


2.31 


12 


90 




2032 


Accti. Receivable 


Yacht Kf9* 


219.438 


2.50 


7 


S9 




2033 


F1 en "0- writer Operator 


Instrument Hfg. 


203*588 


2*50 


12 


79 




2034 


Outreach Cbnmimfty worktr 


Comunlty Servfce 


19S.168 


2*00 


3 


S6 




203S 


Aiiembly Operator 


Radio Kfo. 
Hoblle KSne Kfg. 


726.781 


2.71 


12 


93 




2036 


General Assembler 


806*884 


2*9S 


18 


72 


4 



TABLE 5-6 (continued) 





Iffpioytr Job Titf* 


Type of CCMpeity 


DOT 
C6<k 


Higes 
Ool lers/Hour 


CI ufter 
KL^er 


CI lifter 
Hitch 

In^ex _i 


If 1 a A# 
nUwCr QT 

rf acefocn^ 


1001 


Office Trainee 


Produce Distributor 


2D9.38S 


2*2S 


12 


SS 




3002 


File Cleric 


Insurince 


2D6.3&a 




11 


80 




3003 


ShIpfUter Helper 


Steel Co. 


S06.8S7 


3*79 


IS 


SS 


2 


3O04 


Layout Helper 


Steel tx>* 


BDKSS7 


3>94 


9 


6S 




300S 


Counter Gt r1 


Muffler Shop 


243>368 


2>00 


1 


39 




3006 


Teller Trainee 


Sank 


212.36B 


2*27 


11 


BO 




3007 


Cottputer Clerk Trainee 


Sank 


213.332 


2*27 


12 


S2 




3008 


Office Trainee 


Produce Wholesaler 


209>3dS 


2*00 


12 


93 




^9 


Statec«nt Clerk 


aank 


209. 3&a 


K93 


11 


S2 




3010 


Accounting clerk 


Lvnber Qmipany 


219.48S 


2>SS 


12 


63 




3011 


Auto EUctrlcUn 


Auto ftepair 


72K2S1 


2*S0 


16 


S2 




3012 . 


Truck Driver 


Shipping 


9DS.8S3 


4*43 


S 


6S 


2 


3013 


Seamstress 


Upholstery Shop 


7SS.3dl 


2*40 


16 


72 




3014 


THnmIng Asseicbler 


Upholstery Shop 


7S0.6B4 


2*40 


IS 


73 


2 


301 S 


}4ech«n1c Trainee 


HotOr Repair 


62D.2S1 


2*25 


9 


69 




3016 


Truck Painter Helper 


Awto Body Bepal r 


84S.7S1 


2>25 


IS 


so 




mz 


Doctor Ass 1 Stmt 


Kedtcal Clonic 


D79-37S 


K65 


10 


SS 


2 


3019 


Receptionist 


Kedlcil Clinic 


237.36S 


K6S 


1 


93 




3020 


Electronic Assembler 


Calculating Machines Hfg<. 


726.731 


1.6S 


9 


34 






Shop Worker 


Hold Casting 


709*007 


1*75 


l3 


30 




3022 


HAchlnlst Apprentice 


Hotor Repairs/Supply 


600.260 


2*30 


15 


S6 




3023 


Shop Helper 


Ketal Finishing 


70S-8$S 


2*00 


16 


S2 




3024 


operator Apprentice 


Tank Cleaning 


89K&a7 


3>ao 


18 


S5 




3026 


File Clerk 


8ank 


206*383 


2*oa 


11 


33 





TABLE 5-6 (continued) 



joa 1 


D4>lQy«r Job Title 


Typ« of Coqptnjr 


DOT 
Code 


Wigts 
Dol Urs/Kour 


Cluster 


Iff us^cr 

Matf h 

Jide^ 


HiMer of 
PUceACftts 


4001 


M>ik1n9 Operator 


Electronics Hf9«. * 


g20«7d2 


2.35 


12 


90 




4005 


Parking Lot Attendant 


Ptrk1n9 Lot 


915.078 


2.00 


11 


78 




4006 


File Clerk 


Stvlni) & Loan 


206.388 


2.45 


12 


89 




4007 


File Clerk 


Insurance 


206.388 


2.31 


n 


89 




4008 


File Clerk 


Koipftal 


206.388 


3.40 


11 


69 




4009 


AsieflibUr 


Truck Aiseoibly 


80&.$87 


3.50 


16 


89 




4010 


Ubellng Clerk 


Garment Hfg* 


229.$87 


2.00 


7 


76 




4011 


General Office 


Heat Co. 


209.388 


2.31 


12 


87 




4013 


Clert-Typfit 


Kfg./Olitrlbutor (Labels) 


209.388 


2.25 


1 


$8 




4013 


Clerk'Typlit 


EmploytnentAralnlng Agency 


209.388 


2.56 


12 


82 




4014 


Asienibi 


Auto Asierrbly 


806.687 


4.60 


10 


07 




4015 


Assembler 


Electronics Ht9* 


726.781 


2.55 


12 


85 




4016 


Pre« Operator 


Rubber Products Kfg. 


559.$85 


3.56 


16 


94 




4017. 


H*OJ* Oper*tOr 


Rubber Products Ufs* 


556.782 


3.56/3*71 


16 


92 


3 


401 e 


Kill Op«rttor 


Rubber Products Ht9* 


599.$85 


3.50 


18 


81 


3 


4019 


Hiic. dundler 


Paper Co<. 


920.$87 


4.20 


15 


82 


2 


4oao 


Upholitertr Tr*lnee 


Upholstery 


760 .$84 


3.25 


10 


81 




4022 


Prep./Flnlih Operator 


Film Processing Lab 


976. $86 


2.92 


12 


84 


4 


4024 


Process Equips. Operator 


Electronics Hf9* 


559.835 


2. $8 


16 


87 




4025 


Uit<f erg round Constnictlon 


Ulnderground Cables 


869.384 


3.50 


10 


60 


2 


4026 


Holding Operator 


Electronics Hfg. 


556.885 


2.63 


\$ 


90 




4027 


Fertnentatlon Operator 


Chemical Co<. 


550.702 


4.09 


15 


86 




4328 


Bank Proof Operator 


Oank 


217.3$8 


2.31 


12 


04 




4029 


KIcroMave Aisefnbler 


Microwave Mfg<. 


726.701 


2.60 


12 


06 




4030 


General Ulnery Morker 


Mlnery 


529.$86 


4.07 


17 


90 


2 


4031 


Electrlcti Assembler 


Electronics Kfg. 


726.701 


2.90 


9 


03 




4032 


Electrical Technician 


Electronics Kfg. 


003.101 


3.25 


5 


79 





t 

1 
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The distribution of the JOCL-Cluster correlation coefficients shows 
that 31 of the 110 JOCLs had correlation coefficients less than 
+80 (or + .80). The mean JOCL-Cluster correlation coefficient was 
+81. The range was from +51 to +96. The low value of the mean 
correlation coefficient and the large number of JOCLs with cluster 
match correlations below +80 is troublesome. The introduction 
to the CJHS Cluster register states the following: 



"Beneath each cluster profile in the register, a listing 
of various jobs appears. These jobs were collected and 
then sorted into the clusters such that the correlation 
between the specific job profile and the cluster profile 
would be on the average +.90, and not less than +.80«" 

The Job Cluster Register (dated October 1971) contains 525 individual 
job titles distributed as shown in Table 5-7J Table 5-8 shows the 
cluster distribution of the 110 job profiles obtained as part of the 
SER/Cleff experiments. 

Table 5-8 shows that the 31 job profiles whose cluster match was 
less than +80 account for 43 of the total 142 placements (30.3%)* 
Comparing the two tables, one sees that the percentage distribution 
of job profiles across the clusters is quite different* In the 
experimental sample, almost 32SC of the HO profiles fell under 
cluster #12, whereas in the Cluster Register, this cluster accounted 
for 9.73; of the jobs listed. 

The basic issue here is whether the JOCL-Cluster correlations 
obtained for the experimental job profiles are due to a lack of 
skill in administrating the profiles or are due to "real** differ- 
ences between the .iobs whose profiles were used to create the Cluster 
Register and those obtained as part of this experiment* In other words, 
the inclusion of more job profiles in the data base used to create 
the clusters could lead to different cluster profiles and even to a 
different nuir^er of clusters needed to satisfy the average and mini- 
mum profile-cluster correlation criteria* This is a difficult 



The number of job titles was obtained ly counting each job title/ 
subtitle/business/DOT code line . (See Part I for an 
example of the contents of the Cluster Register.) The ADP-PDS 
final report states that the Cluster Register **is a composite of 
over 750 entry level up to semi-skilled positions that have been 
computer analyzed and organized into 19 basic work clusters*** 
The reason for the difference is not known* 
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TABLE 5-7 



0ISTRI8UTI0N OF JOB TITLES IN THE 
CJHS JOB CLUSTER REGISTER 



Cluster 
Number 


Numbers of 
Job Titles 
{% of Total ) 


Number of ^Job 
Titles with 
DOT Code Assigned 
in Register 


1 


78 


14.9 


34 


2 


8 


1.5 


5 


3 


25 


4.8 


5 


4 


16 


3,0 


3 


5 


19 


3.6 


6 


6 


21 


4.0 


8 


7 


2 


.03 


0 


8 


16 


3.0 


12 


9 


16 


3.0 


9 


ID 


22 


4.2 


6 


11 


21 


4.0 


7 


12 


51 


9.7 


4 


13 


36 


6.9 


20 


14 


18 


3.4 


12 


15 


45 


8.6 


13 


16 


18 


3.4 


15 


17 


44 


8.4 


21 


18 


68 


13.0 


54 


19 


1 


.01 


1 




525 




235 



208 
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7ABLE 5-8 



DISTRIBUTION OF DATA BASE JOB PROHLES BY HIGHEST 
CLUSTER (CORRELATION) MATCH 

(Note: Nomenclature is X/Y where X = number of profiles 
and Y ~ number whose match index is less than +80) 



ERIC 



Clu:>ter # 


1 


SER Agency Number 
2 t 3 


4 


Totals 


Percent 
of 

Total 


of Total 
Below 

+80 


of Each 
Cell 


1 


1 


3/2 


2 


1 


7/2 


6 4 


/ 6 4 


28 6 


2 


0 


1 


0 


0 


"x/O 


0.9 


/ 0 




3 


1 


2/2 


0 


0 


3/2 


2.7 


/ 6.4 


66.7 


4 


0 


0 


0 


0 


0/0 


0 


/ 0 




5 


2 


1 


0 


1/1 


4/1 


3.6 


/ 3.2 


25.0 


6 


4/3 


0 


0 


0 




3.6 


/ 9.7 


75.0 


7 


1/1 


1/1 


0 


1/1 


3/3 


2.7 


/ 9.7 


100.0 


8 


1 


fx 

u 


1 /I 

1/1 


0 


2/1 


1.8 


/ 3.2 


50.0 


9 


1 


1 


3/2 


1 


6/2 


5.4 


/ 6.4 


33.3 


10 


0 


u 


1 / 1 


0 


1/1 


0.9 


/ 3.2 


100.0 


11 


5/2 


0 


4 


3/2 


12/4 


10.9 


/12.9 


33.3 


12 


10/1 


13/4 


4/1 


8 


35/16 


31.8 


/19.4 


17.1 


13 


0 


0 


0 


0 


0/0 


0 


/ 0 




14 


1 


1/1 


0 


0 


2/1 


1.8 


/ 3.2 


50.0 


15 


0 


2 


4/2 


2 


8/2 


7.3 


/ 6.4 


25.0 


16 


0 


1 


3/1 


5 


9/1 


8.2 


/ 3.2 


11.1 


17 


2 


0 


0 


1 


3/0 


2.7 


/ 0 




18 


1 


3/1 


2/0 


4/1 


10/2 


9.1 


/ 6.4 


20.0 


19 


0 


0 


0 


0 


0/0 


0 


/ 0 




Totals 


30/7 


29/11 


24/8 


27/5 


110/31 








Percent 
less than 
+80 


23.3 


37.9 


33.3 


18.5 


28.2 








Number of ' 
clients hire 
into jobs 
whose clus- 
ter correla- 
tion is 
less than +8 


d 

9 

0 


17 


11 


6 


43 








Percent of 
gl||nt data 


25.7 


40.5 


39.3 


16.2 


30.3 


299 
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issue and is not really resolvable. The following points should be 
kept in mind: 

1) The OOCLs administered as part of this experiment were clone 
by five People, four of whom were trained by Mr, Ron Harris, 
Mr, Harris was the other administrator. The 2 X 5 contingency 
table given below shows the number of JOCLs done by each 
of the five administrators which resulted in profiles whose 
cluster correlations were greater than or equal io +80 or 
less than +80, 



JOCL Profile/ JOCL Administrators 

Cluster 



Correlation 


1 


2 


3 


4 


5 


Total 


<,80 


16 


2 


9 


1 


3 


31 


>,80 


25 


7 


32 


7 


8 


79 


Total 


41 


9 


41 


8 


11 


110 



= 4,30 



Since the calculated chi-square of 4,30 is far below the 
necessary critical value of 9,49 (required for signifi- 
cance at the 9B% confidence level), the hypothesis of 
homogeneity between administrators is substantiated. 
This, of course, doesn't mean that the OOCL's obtained 
by the administrators were in fact the "best" for each 
job. 

The JOCLs were obtained over the course of 8 monttts To 
determine if ttiere was a learning curve effect (i,e,, the 
low cluster correlated JOCLs occurred in greater propor- 
tion early in ^he administration time period), the follow- 
ing 8 X 2 contingency table was analyzed. The calculated 
chi-square of 8,7^ is far below the necessary critical 
value of 14,1 (required for significance at the 95% con- 
fidence level)' Therefore', the hypothesis that low cluster 
correlation JOCLs were obtained in greater proportions 
early in the administration period is rejected. 



800 
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CLIENT AND JOB INDICES AND ANALYSIS 



This section presents the following data and analyses: 

1) Statistical measures and frequency distributions of the 
client and the job indices, 

2) Correlation analyses between client and job indices. 

3) Correlation analyses between JOCL book and card sort 
scores. 

4) Correlation analyses between the JOCL combined scores 
(book plus cards) and job DOT code Data« People. Things, 
digits. 

In the experimental data base, there were 110 clients who took both 
SICL booklets. The remaining 32 clients only took the Likes and 
Dislikes (Preference) SICL. Therefore, certain of the analyses to 
be presented will deal only with the 110 clients who took both 
booklets while others will deal with the total experimental group 
of 142 clients. 



5.3.1 Statistical Measures and Frequency Distributions of 
the Client and Job Indices 

The purpose of this analysis was to examine the distributional nature 
of the variable parameters employed in the study. Of primary concern- 
here was to determine how nearly each conformed to the "normal" dis- 
tribution and, in the cases of serious deviations from normality, to 
attempt to find suitable transformations that would achieve at least 
approximate normality for each parameter. 

In advance, it was felt that the Client Adjustment and Job Match 
Indices might be seriously non-normal. Since each is essentially a 
"correlation coefficient," Fisher's "2" normalizing transformation as 
was employed to see If It might provide a more closely normal approxi- 
mation for these Indices, This transformation is as follows: 



2(JMI) = 0.5 In f-TM 



JMI - Job Match Index 
CAI = Client Adjustment Index 



Z(CAI) = 0,5 In ]-mf Z{X) ^ Normal Deviate 



(Standardized) 



ERIC 
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Also, since the "difference" indices are basically "sums of squares," 
they will be distributed as per the Chl-Square distribution. A 
"normalizing" transfomation for this distribution Is provided by 
the following: 

Z(JDI) = ^2(001) - Vzf - 1 
Z{CDI) =v/2(CDI) - v/2f - 1 

Where 

JDI = Job Difference Index 

COI = Client Difference Index 

f = Degrees of freedom for the Index 

(the number of Independent categories upon which 
index is based = 16 for the Cleff system) 



Thus, the transformed difference indices become: 
Z(JDI) =\/2(JDI) - ^ 

Z(CDI) = y2(CDI) - S 

Table 5-9 presents the pertinent condescrlptlve statistics for each 
of the variables of interest, both with and without the employment 
of the above transformations. 

It is seen that In all cases except the case of the ODI for the 32 
clients who only completed the Preference SICL, the use of the normal- 
izing transformation did result In a significant reduction in dis- 
tribution skewness towards the "Ideal" normal distribution value of 
zero. 

Figures 5-1 and 5-2 show the frequency histograms of the Client 
Adjustment and Job Match Indices. The Client Adjustment histogram 
Is based on the 110 observations. The total Job Match Index histo- 
gram is based on all 142 cases. In addition, the histogram for the 
110 clients who took both booklets Is shown within Figure 5-2 as 
the diagonal slashed portion of the total histogram. Table 5-10 
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TABLE 5-9 

CONOESCRIPTIVE STATISTICS FOR CLIENT AND 008 MATCH INDICES 



No. of Observed Range 
Observa- Standard 



Parameter 


tions 


Min. 


Max. 


Mean 


Oeviati on 


Skewness 


Not Trans- 














formed 














JMI 


142 


-.64 


.83 


.291 


.291 


-.578 


001 


142 


805 


4919 


2158 


895 


.876 


CAI 


110 


-.55 


.95 


.371 


.374 


-.633 


CO I 


110 


116 


4000 


1273 


1 at 


1 118 


JMI 


no 




.83 


.292 


.26 


-.489 


JO I 


no 


805 


4919 


2075 


848 


1.168 


JMI 


32 


-.64 


.770 


.288 


.374 


-.627 


JOI 


32 


872 


4358 


2441 


1004 


.100 


Trans- 














formed 














Z(JMI) 


142 


-.758 


1.188 


.330 


.344 


-.178 


Z(JOI) 


142 


34.56 


03.62 


58.77 


13.32 


.340 


Z(CAI) 


110 


-.618 


1.832 


.472 


.501 


.108 


2(C0I) 


110 


9.66 


83.88 


42.78 


14.55 


.356 


Z(JMI) 


110 


-.448 


1.188 


.327 


.311 


-.007 


Z(JOI} 


110 


34.56 


93.62 


57.62 


12.65 


.509 


Z{JMI) 


32 


-.758 


1.020 


.340 


.446 


-.405 


£(JOI) 


32 • 


35.19 


87.79 


62,73 


14.97 


-.231 
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Figure 5-K Frequency Histogram of the 
Client Adjustment Index 

• 1 10 Observations 

• Class Interval Oefined as 
Lower Limit < CAI^ Upper 
Li mit 
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Figure 5*2. Frequency Histogram of the Job 35"- 
Match Index 

♦ 142 Observations 

♦ Class Interval Defined as 
Lower Limit < JMI Upper 
Limit 
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TABLE 5-10 



CLIEKT AOOUSTH£NT AND JOB MATCH INDICES 
FREqUENCr HISTOGRAM TAILY SHEET 



Indice Value Client Adjusttnent 

Grtater L«ss 

Than or Than or Cum.Freq. 
faual to EQual to Frtanpncv f Percent) 



91 


100 


3 


(100.0) 


81 


90 


6 


( 97.3} 


71 


80 


14 


( 91.8) 


61 


70 


11 


( 79.1) 


51 


60 


18 


( 69.1) 


41 


SO 


5 


( 52.7) 


31 


40 


13 


( 48.2) 


21 


30 


7 


( 36.4) 


11 


20 


10 


( 30.0) 


1 


10 


3 


( 20.9) 


-9 


0 


4 


( 18.2) 


-19 


-10 


1 


( 14.5) 


-29 


-20 


8 


( 13.6) 


-39 


-30 


4 


( 6.4) 


-49 


.40 


2 


( 2.7) 


-59 


-50 


1 


{ 0.9) 


-69 


-60 


0 




-79 


-70 


0 




-89 


-80 


0 




-100 


-90 


0 





Job Hatch Index Total 
Both One 

SICLs SICL Cum^Frtq, 

Frequency freouftncv Frequency f Percent) 



0 


0 


0 






0 


1 


(100.0 




4 


8 


( 99.3 




5 


13 


( 93.7 




1 


7 


( 84. & 


21 


3 


24 


( 79.6 


20 


5 


25 


( 62.7 


14 


3 


17 


( 45.1 


11 


2 


13 


{ 33.1 




1 


10 


( 23.9 




0 


9 


( 16.9 




3 


4' 


( 10.6 


2 


3 


5 


( 7.7 


3 


1 


4 


( 4.2 


1 


0 


1 


( 1.4 


0 


0 


0 


( 0.7 


0 


1 


1 


( 0.7 


0 


0 


0 




0 


0 


0 




0 


0 


0 
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shows the frequency histogram tally sheet used to generate the 
histograms. The upper and lower class boundary definitions are 
shown in this table. The cumulative frequency percentages for 
the CAI and all the JMIs (one or both SICLs) are also shown. The 
data shows that 18.2% of the 110 clients who took both SICLs had 
CAIs less than or equal to zero. For the 142 clients, 16.9% had 
match indices less than or equal to zero, and 79.6% had match in- 
dices less than or equal to +50. (The range of the indices was 
given in Table 5-9.)* 



5.3.2 Correlation Analyses Between Client and Job Indices 

Correlation analyses were conducted to determine if the index mea- 
sures were correlated or if they could be regarded as independent 
measures. Table 5-11 summarizes the results of these correlations. 
The results show that the Job Match and Difference indices and the 
Client Adjustment and Client Difference indices are significantly 
inversely correlated. The correlation of the Job Match and Client 
Adjustment indices supports the conclusion that there is no signi- 
ficant correlation between these two indices. These results are 
what one would expect given the manner in which these indices are 
obtained. 

The scatter diagram showing the individual client adjustment and 
job match indice pairs is shown in Figure 5-3. The circled and 
uncircled points are in toto the 110 cases for which the CAI is 
available. The circled points represent those cases where the 
individual was still working at the final re+-ention followup time 
point, i.e., December 10, 1974. The diagram visually shows what 
the correlation of these two indices yielded. 

The scatter diagram has been divided into 9 areas based on the values 
of the two indices. This is shown in Figure 5-4. The nine areas 
or cells have been defined as follows; 



Quadrant 



Areas (Cells ) 



I CAI > 0 
JMI > 0 



0 < CAI< +.25 
0 < JMK +.50 



2. 



+ .25 < CAI < 1.0 
0 < JMI < +.50 



3. 



+ .25 < CAI 5 1.0 
+.50 < JMI ^ 1.0 
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TABLE 5-n 

ANALYSIS OF CORRELATION BETWEEN INDEX MEASURES 







Correlation, r 




Description 


Number of 
Observati ons 


Data 
Transformed 


Data Not 
Transformed 


Significance 
Probability 


1 . Job Match & Job 

Difference 
Indices 










a. All Cases 


142 


-0.724 


-0.721 


.001 


b. Clients Who Took 
Both SICLs 


110 


-0.676 


-0.677 


.001 


c. Clients Who Only 
Took Preference 
SICL 


32 


-0.870 


-0.852 


.001 


2. Client Adjustment 
and Client Differ- 
en:e Indices 










a. Clients Who Took 
Both SICLs 


110 


-0.814 


-0.774 


.001 


3. Client Adjustment 
and Job Match 
Indices 










a. Clients Who Took 
Both 8ooks 


110 


.047 


0.035 


Not 
Significant 
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Figure 5-3. Scatter Diagram 
JMI vs. CAI 

• Circled Points Are for Cases 
Still Working as of 
December ID, 1974 

• Solid Points Are for Cases 
Terminated Prior to 
December ID, 1974 

• IID Dbservations 




Figure 5-4. Distribution of Paired Observations: 
Client Adjustment Index (CAI) and 

• no Observations 

• Percentages based on 110 cases 



© 



5(4.5%) 



JMI 



100 

2(1.8%) 

® 

^ SO 


3(2.7%) 

t® 


14(12.7%) 

© 


13(11.8%) 

© 


11(10%) 

© 


51 (46.3%) 

© 



-100 



2(1.8%) 



© 



9(8.2%) 



,CAI 



© 
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Quadran t 



Areas (Cells) 



4. 0 < CAI < +.25 
.50 i JMI i 1.0 



II CAK 
JMI > 



0 
0 



5, 



1, CAI < 0 
+.50 < JMI i +1.0 



6. 



CAK 0 
0< JMI< +.50 



III CAK 
JMIi 



0 
0 



7. 



Same as quadrant 



IV CAI> 
JMI< 



0 
0 



8. 



0 £ CAI £ +.25 
JMIi 0 



9.+.25£CAI £ 1.0 
JMI i 0 



As was discussed earlier in this report, there was a cutoff criterion 
established for the New Jersey experiment which was to govern the 
referral process. If the client's adjustment index was greater 
than or equal to +.25 and if the client-job match index was greater 
than or equal to +.50, then the client was "eligible" for referral 
and the job was "eligible" for this client to be referred to. The 
distribution shown in Figure 5-4 indicates that 14 client-job matches 
out of the total of 110 fell into this range (cell 3 of quadrant 1). 
This is only 12.73S of the 110 client-job matches where the client 
took both books. There were a total of 74 clients whose adjustment 
indice was greater than or equal to +.25 (cells 2,3,9). This is 
67,3% of the 110 clients. In addition, there were a total of 19 
whose job match index was greater than or equal to +.50 (cells 3,4, 
and 5), Of tbe 142 clients, there were a total of 29 whose job match 
index was greater than or equal to +.50. This is 20.4% of the total 
142 clients. Thus, the data clearly shows that the vast majority of 
client-job matches would not have met either the job match index 
criterion above or the combined client adjustment and job match 
criterion. In addition, one sees that almost one-third of the 
people would not have met the referral "eligibility" criterion based 
on their client adjustment indice. What this means is that if the 
criterion that was established for the New Jersey experiment had been 
applied to the SER experiment than an entirely different set of job 
placements would have been obtained. Obviously, the attainment of 
such placements would have depended on the availability to SER of 
suitable job openings. 
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TABLE 5-12 

CORRELATION BETWEEN JOCL BOOK 
AND CARD DIMENSION SCORE 



Component 




0.770 


\ 


0.335 




0.758 




0.504 


T 

^5 




T 

^6 


U.4o/ 


T 

^7 




T 


U.334 




0.432 


h 


0.428 




0.383 


''4 


0.504 




0.353 


h 


0.555 


^3 


0.707 


^4 ' 


0.386 
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It should again be noted that the manner in which Ultrasystems 
instructed the respondent to sort the OOCL cards is different 
than that specified by the instructions as stated in the OOCL 
itself. This was discussed earlier in this report (see Section 
4). 



5.3.4 Correlation Analyses Between OOCL Combined Scores 
and the Oob OPT Code Oata. People Jhings.Ojgits 

The purpose of this analysis was to determine if there is a rela- 
tionship between the OOCL dimension scores and the Oata, People, 
Things digits of the OOT codes assigned to the jobs that were pro- 
filed. The 6-digit OOT code was assigned by Ultrasystems based on 
the job description obtained as part of the administration of the 
OOCL. These assigned OOT codes were shown in Table 5-6. 

This analysis was conducted using the total of 118 OOCLs obtained 
(see the discussion in Section 5.3.3), 

An Initial set of correlation analyses was performed within the OOT 
Data, People, Things digits for the 118 jobs. The results are shown 
in block ® of Table 5-13. All the correlation coefficients were 
found to be significant at the 99% confidence level. 

Thus, the "Data" and "People" digits are seen to be positively 
correlated over the set of jobs used, while the "Things" digit 
was inversely correlated with both the "Oata" and "People" digits. 

i|owevet,^_dJ!lCfiJ;J^^ was-done^itfe -otvl-y- *he-l-lS-j«bs-that 

were profiled, there is no way of knowing to what degree these re- 
sults would hold across all the DOT codes. 

Correlation analyses were then conducted between the OOT Data, People 
Things digits and the total sum of the OOCL dimension scores in the 
three dimension categories Things, People, and Ideas. The results 
are shown in block (2) of Table 5-14, What has been done here is to 
calculate for each OOCL the sum of the 8 dimension scores relating 
to Things, the 4 dimension scores relating to People, and the 4 
dimension scores relating to Ideas. All three of these correlations 
were found to be statistically significant at the 99.9% confidence 
level . 

Thus, In each case, the DOT digits are seen to be inversely corre- 
lated to the Cleff rating for the corresponding job characteristic 
skill level. This would be the expected result, since the OOT 
rating Is lowest for an "increased skill level requirement, while 
the opposite is true of the Cleff rating scales. 
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TABLE 5-13 



CORREUTIONS BETWEEN DOT CODE DATA. PEOPLE. THINGS 
AND THE cans JDCL DIMENSION SCORES 



0 DOT Digits Correlated 



ERIC 



DATA (4th digit) vs. PEOPLE (5th digit) 0.414 
DATA (4th digit) vs. THINGS (6th digit) -0.361 



PEOPLE {5th digit) vs. 


THINGS (6th digit) 


-0.247 


(2) DOT Oiqit vs. aoCL Total Score 


r 


DATA (4th digit) IDEAS (set of 4) 
PEOPLE {5th digit) PEOPLE (set of 4) 
THINGS (6th digit) THINGS (set of 8) 


-0.436 
-0.620 
-0.365 


® DOT Digit vs. aoCL Component 


r 


DATA (4th digit) 


^1 


-0.178 


II 


h 


-0.138 


II 


h 


-0.562* 


II 


h 


-0.228 


PEOPLE (5th digit) 




-0.532* 


II 




-0.139 


It 




-0.407* 


11 




-0.430* 
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The above analyses were also performed by individual JOCL dimension 
for the Ideas and People characteristics in order to determine which 
components were most correlated with the corresponding DOT digit 
rating. The results are shown in block (3) of Table 5-13. 

The correlation estimates followed by * are the only ones found to 
be statistically significant at the 99% confidence level. Thus, 
the DOT Data digit appears to be correlated only to JOCL component 
U, at least in a major way, while three of the four JOCL People 
components were found to be significantly correlated to the DOT 
People digit. 

As mentioned previously, however, a wider range of job profile data 
sets would have to be analyzed in order to verify and generalize 
these results. 



5.4 RETENTION DATA AND ANALYSES 

Before presenting the retention data obtained and the analyses 
performed, the following points are presented as a brief review 
of the key aspects of this part of the analysis: 



1) The retention of the 142 people included in our sample 
was obtained by telephone contact with the respective 
employers. These telephone contacts were done at four 
time points, i.e., December 31, 1973, February 28, 1974, 
May 31, iq7d. and n prwnhpv' in, ^<i^A 

2) The employer was asked if the person was still working and 
if the response was No then the date terminated and the 
reason for the termination was obtained. No questions 
were asked regarding job or salary changes. 

3) The sample of 142 people consisted of one group Of 110 
people who c(»iipleted both SICL booklets and 32 peoole who 
only completed the preferences (Likes and Dislikes) SICL. 

4) At the final retention followup date {December 10, 1974), 48 
of the 142 people (33.8%) were still employed witn their 
original employer. The 142 people were hired into their 
respective jobs over a seven-month time period, i.e., from 
early March 1973 to early October 1973. Thus, the maximum 
and minimum retention periods could be from approximately 21 
months to 14 months, respectively. 
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5) No information was obtained from the people themselves nor 
was any other data regarding their job perfonnance other 
than their retention, obtained. 

The subsections to follow will present the following data and 
analyses: 

Subsection Data and Analyses Presented 

5.4.1 Hire* termination, and days worked data distributions 
for the sample, 

5.4.2 Statistical analyses of the differences between client 
and job match indices for client groups obtained by 
grouping those clients who worked less than or equal 
to "X" days and those who worked longer than "X" days. 
The analyses were done for 5 values of "X"; e.g., 30, 
60» 90, 120, and 199 days. In addition, the same 
analyses were done between those clients who terminated 
prior to Decenber 10, 1974 and those who were still 
working as of this date. 

These analyses are referred to as the Truncation of 
Time Worked Analyses. 

5.4.3 Statistical analysis of the relationship between the 
job match index and the subsequent retention done from 

the standpoint of a life testing proMem. This analy- 

Sis, k'ef^rrea zo as tne "exponential analysis," uses 
a mean-tlme^etween-failure variable obtained by summing 
all the days worked by a group of people (whose job 
match Index lies In a certain range) and dividing this 
sum by the number of these people who terminated prior 
to December 10» 1974. 

5.4..4 Multiple regression analyses using days worked as the 
dependent variable and the client adjustment and job 
match indices as the Independent variables. Two analyses 
were done, one of which Involved the 71 people who took 
both SICLs and vrtio terminated prior to December 10» 1974, 
The other one utilized all 110 cases where both SICLs 
were taken and assigned a randan future termination time to 
the 39 people of this group who were still working as 
of December 10* 1974. In addition* the effect of the 
restriction on known days-worked-tlll-tennlnatlon was 
estimated using a statistical methodology. 



5.4.5 Temlnatlon reason data and distributions for the sample 
of 94 people who terminated prior to December 10, 1974. 
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5.4.1 Hire. Termination^ and Days Worked Data Distributions 

Figure 5-5 shows the frequency histogram of the date of hire for the 
142 people in our sample. The frequency histogram uses a tvw 
calendar week class interval. The earliest hires occurred in the 
two-week period March 3 through March 14. 1973, and the last hires 
occurred in the two-week period September 22, through October 6. 
1973, Of the total 142 hire dates, 75 or 52,8% occurred on or 
before June 9, 1973. The diagonally slashed portion of the histo- 
gram shows the hire date distribution for the 110 clients who took 
both SICLs, 

Figure 5-6 shows the frequency histogram of the calendar days 
worked till termination for the 94 people who terminated prior to 
December 10. 1974. The class interval is 30 calendar days. The 
diagonally slashed area of the histogram shows the distribution for 
the 71 people who terminated prior to December 10» 1974 and who took 
both SICLs. The days-worked-till-tennination histogram shows in a 
general sense an exponential distribution. Included in Figure 5-6 
is the frequency histogram for the calendar days worked for the 48 
people who were still working as of the last retention foUowup time 
of December 10, 1974, Again, the diagonally slashed area shows the 
distribution for those people who took both SICLs. If one visually 
moves the days worked distribution for those 48 people who were 
still working at the time of the last foUowup down to the bottom 
basicca. one sees that the known distribution of days worked is. 
broadly speaking. U-shaped. 

Figure 5-7 shows the cumulative frequency and percentage cumulative 
frequency distributions for the 94 people who terminated prior to 
December 10, 1974, The solid ogive shows the cumulative frequency 
and the dashed ogive the cumulative percentage frequency. One sees 
that 50% of the total terminations worked less than or equal to 90 
days. Thus, of the total 142 people whose retention was tracked, 
48 or 33.8% worked less than or equal to 90 calendar days. Again, 
of the total of 142 people, the figure shows that 71 or 50% worked 
less than or equal to 180 calendar day£i. 

Table 5-14 shows the frequency histogram tallies used to generate 
the figures. 
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Figure 5-5. Frequency Histogram of the 
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figure 5-6. Frequency Histograms: 

• Calendar Days Employed Till 
Termination {94 observations) 

• Calendar Days Employed for 
Clients Still Working as of 
12/10/74 {48 observations) 

Note: Diagonal Slashed Area Is for Clients 
Who Took Both SiCLs. 
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TABLE 5-14 

HteqUENCY KISTOSttAK TAllIES FOR OAYS WRKEO 



Number of Days 


Client 


Client 








For Those People 


Took 


Took 








Who Temin*ted Prior 


&oth 


1 SICl 




Cum. 


Cum. 


to ueMoiher 10. 1974 


SICLs 


(Pref) 


Totdl 


Totdl 


Percent 


0 


30 


23 


S 


2a 


26 


29.8 


31 


60 


7 


3 


10 


38 


40.4 


61 


90 


6 


3 


9 


47 


50.0 


91 


120 


5 


2 


7 


54 


57.4 


121 


150 


8 


2 


10 


64 


68.1 


151 


180 


5 


2 


7 


71 


75.5 


181 


210 


4 


2 


6 


77 


81.9 


211 


240 


2 


1 


3 


60 


85.1 


241 


270 


1 


0 


I 


81 


86.2 


271 


300 


1 , 


4 
1 


2 


83 


00. J 


301 


330 


0 


1 


1 * 


84 


09. 4 


331 


360 


1 


0 


1 


ac 
09 


90.4 


361 


390 * 


3 


I 


4 


89 


94.7 


391 


420 


1 


0 


\ 


90 


95.7 


421 


450 


1 


0 


1 


91 


96.8 


451 


480 


1 


0 


1 


92 


97.9 


481 


510 


0 


0 


0 


92 


A ^ A 

97.9 


511 


540 


1 


0 


1 


93 


AA A 

98.9 


541 


570 


0 


0 


0 


93 


98.9 


571 


600 


\ 


0 


1 


94 


100.0 




Tot«l 


71 


23 


94 






Oays Worked for 












Those People Still 












Workin9 as of 












December 10. 1974 












451 


480 


4 


1 


5 


5 


10.4 


481 


510 


5 


3 


8 


13 


27.1 


511 


540 


7 


0 


7 


20 


41.7 


541 


570 


5 


0 


5 


2S 


52.1 


571 


600 


13 


3 




41 


85.4 


601 


630 


4 


2 


6 


47 


97,9 


631 


' 660 


1 


0 


1 


48 


100.0 



Totals 39 9 48 



Number of Days 


All TenniMtlons 


Terminations Wio Took 


For Those People Who 






8oth SICLs 


Teminated Prior 


Nunber of 








to Oecember 10, 1974 


TeriRimtions 


CumuUtive 


Number 


Cumulative 


0-30 


28 


28 


23 


23 


31 - 60 


10 


38 


7 


30 


61 - 90 


9 


47 


6 


36 


81 - 120 


7 


54 


5 


41 


121 - 199 


20 


74 


14 


55 


200 - 299 


9 


83 


7 


62 


300 - 399 


6 


89 


4 


6fi 


400 - 499 


3 


92 


3 


69 


500 - 599 


2 


94 


2 


71 
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5.4.2 Analysis of Cleff Client and Job Match Indices 

and Fixed Truncation Levels of Job Retention Time 



The purpose of this analysis was to determine if there were certain 
levels of the client and job match indices which served to differen- 
tiate between significant shifts in job retention times. 

The job retention times selected to be investigated for this analysis 
were as follows: 

Worked less than or equal to, conpared to, worked more 
than 

A) 30 days 

B) 60 days 

C) 90 days 
D 120 days 
E) 199 days 

Tables 5-15 and 5-16 present the pertinent data that were used for 
each of the above analyses. Table 5-15 shows the 'data and analyses 
done using the actual values of the indices. Table 5-16 shows the 
corresponding data using the transformed indices. The entire sample 
of 142 was used for the JMI and ODI analyses. The corresponding 
sample for the CAI and CDI analyses was limited to the 110 clients 
who completed both booklets. The student "t" test was applied to 
the worked-less-than-or-equal-to and worked-more-than index averages 
resulting fron the two sets of cases obtained from each truncation 
level. 

The data shown in the table indicate that there is a statistically 
significant relationship between the job match index and the proba- 
bility that the person will remain anployed over the initial 120 
calendar days frcHn the date of hire. 

The optimum level of truncation for the JHI appears to occur at about 
60 days. As can be seen in the two tables, the value of "t" for the 
t-test for the JHI is maximum at this level. Also, it is the only 
level at which all 4 indices are judged simul taneously to be signi- 
ficantly different in both tables, i.e., transformed Indices and 
actual indices. 

The CAI and CDI indices appear to pass through an initial optimum 
also at 60 days; however, there is also a long-term effect which 
becomes even more significant at 199 days and is maximal at the time 
of the last employment status check. This is more clearly seen by 
the t-tests, which compare all "terminated" versus all "still working" 
as of December 10, 1974, as shown in Table 5-17. 
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TUU AHALYStS Or CLEFf SYSTEH SCOftCS VERSUS LEHGIH OF TIW 

WRK£0: TIK WMtKCO CAIECOftmO ]H OlSCttETE ttlCROtEHTS 
nxEO IRiACAYtON POtHT^ 

m%t (\) Th« d«tt iNwn for the CA1 «n4 CO] Is 
for no cHtnts. 

(2) The d«tA shown for the 0H1 «nd 001 ts 
for 142 cUcfits. 

(3) Indicts Are actual values 



Oob ftettntlon Times 
(tn 0*¥iJ 


CA1 


CDt 


OKI 


001 


(A) 

0-30 


X 

i 

n 


26.4 
36.6 
23 


1512.4 
833.9 
23 


14.2 
31.6 
28 


2274.0 
949.6 
28 


>30 


X 

s 
n 


39.9 
35.9 
87 


1210.7 
721.2 
87 


32.8 
26.9 
114 


2129.2 
870.6 
114 




t 

*Sfg. 


1.60 


-1.73 
+ 


3.16 
+++ 


•0.77 





X 


25.3 


1438.2 


13.7 


2396.6 


0.60 


s 


30.5 


798.0 


32.0 


1071.2 




n 


30 


30 


38 


3e 




ic 


41.5 


1193.3 


34.8 


2070.5 


>60 


i 


35.1 


726.5 


26.3 


810.9 




n 


ao 


80 


104 


104 




t 


2. to 


-1.85 


3.99 


-1.94 




519^ 


+ 


+ 


++* 


+ 







33.4 


1356.9 


17.3 


2364.3 


0-90 


i 


35.7 


744.9 


31.2 


1038.6 




n 


3$ 


36 


47 


47 




X 


36.9 


1233.3 


35.0 


2055.5 


>90 




36.1 


746.6 


26.2 


S05.0 




n 


74 


74 


95 


95 




t 


0*75 


.0.82 


3*55 


-1.95 




Sig. 






+++ 


+ 





X 


31.1 


1340.5 


19.7 


2Z96.5 


0-120 


i 


36*2 


758.8 


30.5 


iooa.0 




n 


41 


41 


54 


54 




£ 


40.7 


1234.1 


34.9 


2072.6 


>120 




34.8 


739.3 


26.3 


8G8.9 




n 


69 


69 


ee 


6d 




t 


1.35 


.0.72 


3.14 


-1.46 




Sig. 






++ 





X 


27.1 


0- z 


37.1 


n 


55 


X 


47.2 


> 199 i 


3S.4 


n 


55 


t 


2.91 


Sig. 


++ 



1447.6 


25.6 


2154.4 


794.0 


29.2 


926.4 


55 


74 


74 


1100.0 


32.9 


2161.4 


704.3 


26 8 


852.7 


55 


60 


68 



-2.43 1.55 0.05 

++ - * 



Significance C<Kle; " * difference botHeen ncini mt significant 

* ' I " i^nlfkAnt At 95X cMfldence level 

■ * * * * » It ■ 

♦++ ».« • 
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TAILE 5-16* AHALTS1S OF CLEFF STSTCH SCORES VERSUS LENGTH OF TlHE 
1I0RKE0: TINE HORKEO CATEGORIZED IN OISCRETE IMCREHEHTS 
0*E*. FIXEO TRUftCATlOH POINTS) 

Note: (t) The dm Shown for the CAI and COI Is for 
no cUents* 

(2) The dm shwm for the M and 001 Is for 
U2 clients* 

(3} Indices are transformed values* 



Oob Retention Times 
hn (I4¥i) 


ZttALL 


ZiCOI) 


Z(ON]) 




0-30 




s 
n 




*303 
*42l 
23 


47*52 
14.70 
23 


*154 
*351 


60*52 
13*67 
28 


>30 




s 
n 




*517 
*47l 
87 


41*53 
13*51 
87 


*37d 
*317 
114 


S8*34 
13*03 
114 




t 

Sl9 






1*^ 
t 


-1*86 
t 


3.28 
++t 


*0*79 


0.60 




s 
n 




*298 
*453 
30 


47*17 
14*44 

30 


*148 
.352 
36 


G2*0l 
15*24 
36 


>60 




s 
n 




*537 
*464 
80 


, 41*14 
13*52 

eo 


*402 
*314 
104 


57*5B 
12*33 
104 




t 

sig 






2.42 

+4 


-2*05 
t 


4.13 
m 


-1.78 





.421 


* 44.40 


*191 


61*62 


s 


.448 


13*94 


*34d 


14*92 


n 


36 


36 

• 


47 


47 


r 


*497 


42*00 


*405 


57*36 


s 


.467 


13*70 


*312 


12.23 


n 


74 


74 


95 


9$ 




0*81 


-0.86 


3*70 


-1*81 



sl9 





.3^ 


44*03 


*220 


60*63 


s 


*480 


14*35 


*347 


14.65 


n 


41 


41 


54 


54 


7 


*516 


42.04 


*404 


57*62 


s 


.450 


13.44 


*310 


12*21 


n 


69 


69 


SB 


68 




1*30 


-0.73 


3*28 


-1*32 



s19 . , m 





*34l 


4^*10 


*292 


58*56 


s 


.455 


14.34 


.343 


13*72 


n 


55 


55 


74 


74 


7 


*603 


39*47 


*380 


58*99 


s 


.4&6 


13.22 


*304 


12*55 


n 


55 


55 


68 


^ 66 




2.98 


•^2*52 


1*61 


0*19 



Sig ++ ++ 



S19ft1ftci(W« Code! - • difftrtnce bttmn Mins not stoMflcint 

t • * • • significant «t 95X confidence livil 

^ ^ • • • h h It h 
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TABIE 5-17. ANALYSIS OF SCORES (HJOICES) FOR THOSE PEOPIE STlll 
WORKING AS OF DECEMBER 10, 1974 AND THOsE WHO 
TERMIHATED PRIOR TO THIS TIME 



Statos as of 
12-10-74 


CAI 


COI 


OHI 


001 


All 


X 


29.1 


1459.3 


26.1 


2201 .5 


Terminated 


s 


37.3 


786.1 


28.9 


933.4 




n 


71 


71 


94 


94 


All 


J 


. 51.8 


936.1 


34.9 


2072.1 


Still 




35.3 


701.1 


27.0 


827.2 


Working 


n 


39 


39 


4£t 


48 




t 


3.11 


-3.47 


1.75 


-0.81 




slg 


++ 


+++ 


+ 





ZfCAlJ ZfCDl) Z(OHI L ZfJDI ) 



All 


7 


.363 


46.47 


T 

.296 


59.29 


Termi nated 


s 


.300 


14.62 


.356 


14.09 




n 


71 


71 


94 


94 


All Still 


x 


.670 


36.15 


.407 


57.75 


Working 


s 


.458 


11.62 


.295 


11.75 




n 


39 


39 


48 


48 




t 


4.24 


-3.80 


^ 1.86 


-0.65 




sis 


+++ 


+++ 


+ 





(See Table 5U6 for sisnlflcance codes.) 
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A11 the people who terminated prior to this tinie had an average CAI 
of 29.1 compared to an average of 51.8 for those still working. The 
JMI comparison* although significant* had a much smaller difference 
(26.1 versus 34.9 for the still -working). 

Thus, the data Indicates that although both CAI and JMI are related 
to job retention, it is the CAI which relates to long-term employment 
potential while the JMI is the important measure of surviving the 
initial period of employment, particularly during the first three 
months. In addition, the analyses show that the use of the actual 
Indlce values as opposed to the transformed values yields essen- 
tially the same results. 

Figure 5-8 shows, graphically, the means of the CAI and JMI for the 
two client groups over the five values of calendar days worked. The 
figure shows the emerging divergent pattern of the CAI between the 
worked-greater-than and worked-less-than client groups when the 
number of days ts greater than (or less than or equal to) 90. 

Figures 5-9 through 5-13 show a series of scatter diagrams of the 
client adjustment and job match tndlce pairs. All these figures 
show the scatter diagram for the 110 people who took both SICLs. 
The first diagram (Figure 5-9) differentiates between those clients 
who terminated prior to Decen^er 10, 1974 (the points) and those who 
were still working at that time (the circled points). The next four 
figures Identify in a cumulative manner the cases who terminated 
after working less than or equal to 30, 60, 90, and 120 days, respec- 
tively. These cases are Identified by the cross over the point. 
Figure 5-14 shows the frequency counts of the number of cases and 
the number of terminations in the nine cells defined In Section 5.3.2 
(note: Figure 5-9 was presented In this section as was part of 
Figure 5-14). The slashed cells Identify those ranges of CAI and 
JMI values where all the clients which had Indlce values In this 
range terminated prior to Decerhber 10, 1974. 

Figure 5-14 and the scatter diagrams graphically illustrate the 
relationships shown In Tables 5-15 through 5-17, As an example, 
the data shown In Figure 5-14 shows that for those 74 clients whose 
CAI was greater than or r^qual to -^25 (cells 2, 3, and 9), 42 (56.8%) 
tennlnated prior to Dectniber 10, 1974, whereas for those 36 whose 
CAI was less than -^25 the number who terminated was 29 (80.6%). 
This way of looking at the relative retention masks the actual length 
of employment due to the wide range of hire dates shown earlier. 
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Figure 5-8, Hean VAlues of the CAI and 
OHI for Groups Employed Less 
Than or Equal To or Greater 
Than "X" Days 

Note: Number of Observations at Values of 
"X" Are Shown in Parentheses, 
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1^, 




j O^' e^^AL TO *X' OAVS 



I 

4- 



no 



( 

Figure 5-9. Scatter Diagram Job Match 
Index and Client Adjustment 
Index 

• no Observations 

• Terminations Are Points 
(71 observations) 

• Still Working as of 
December 10, 1974 Are 
Circled Points (39 obser- 
vations) 




Figure 5-10. 



Scatter Diagram JMI and 
CAI 

• Terminations Employed 
Less Than or Equal To 
30 Days Marked by X 
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Figure 5-lU Scatter Diagram JHI and 
CAI 

• Terminations Snployed 
Less Than or Equal To 
60 Days Are Harked by X 




Figure 5-12. Scatter Diagram OMI and 
CAI 

• Terminations Employed 
Less Than or Equal To 
90 Days Are Marked by X 




Figure 5-13. JHI and CAI 

• Tenninations Employed 
Less Than or Equal To 
120 Days Are Marked by 




Figure 5-14. 



Distribution of Paired Observations: 
Client Adjustment Index (CAI) and Job 
Match Index (JHI) 

• Total of 110 Observations 

• As of December 10, 1974, 71 Had Terminated (64.5%) 

• Numbers Above Line in Each Cell Are the Number 

of Observations and the Percent of Total Observations 

• Numbers Below Line in Each Cell Are the Number of 
Terminations and Percent of Cell That Terminated 
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5.4.3 Job Retention Time and Job Match Index: 
Exponential Analysis 

An alternate way of viewing the job-retention time parameter is to 
consider it fron the standpoint of a life-testing problem. Thus, 
one considers the length of time anployed on a job in a similar way 
to the classic "lightbulb" test where a given nurflber of units are 
turned on at a particular time and one records the time-to-failure 
(or. in this case, time-to-job-termination) as a result. After a 
sufficient number of terminations have been observed, the exponen- 
tial distribution may, for example, be fitted to the observed data 
as a method of relating the various indices to job retention. 

Table 5-18^ presents a tabulation of the observed job-retention 
times as of December 10. 1974 {the terminations are shown in 
italics) classified as a function of JMI intervals of 10%. For each 
interval, the estimated exponential parameter {job-retention time 
per tennination) was calculated. For each interval, this is done by 
computing the total time worked (both for the "terminated" and "still 
working") and dividing this total by the number of terminations ob- 
served for the interval. {Figure 5-15 shows the exponential param- 
eter histogram.) 

As can be seen, the average retention time does appear to increase 
significantly as JMI increases. The observed increase is not uni- 
form, however, due to the small cell sizes. A gross significance 
test may still be applied to the data in order to test the increase 
by combining some of the adjacent intervals in order to increase 
sample size and hence the precision of the retention-time estimates. 

For example, consider a cut-point at JMI « 0, which results in the 
following canbined retention-time averages: 



No. of Average Retention "F" 

JMI I, Pays Worked Terminations Time (Days) Ratio Sig. 

^0. 33.895 76 446.0 ^ ^ 

< D 4,438 18 246.6 



Note: The class interval boundaries utilized in this analysis are 
greater than or equal to the lower class limit and less than the upper 
class limit. Thus, the class intervals used ars not the same as was 
shown in Figure 5-2 and Table 5-10. 
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TABLE 5-18. EXPONENTIAL ANALYSIS: OBSERVED RETENTION TIME (IN CALENDAR DAYS) 
VERSUS JOB MATCH INDEX 

Note: -Terminations shown in italics 

-Retention as of December 10, 1974 



m MTCK IKOEX 



-,7 



-.6 



-.4 



^.3 


-.2 




D 




2S 


6 


0 


6 


IB 


39 


U 


306 


$ 


6 


204 


BO 


£B4 


30 


27 


582 


596 


582 


12 


5J 




573 




14 


?S 








e? 


1S4 








141 


1?B 








533 


449 










589 



I 

ir ^ 



CO 



.1 



.2 



21 
3 
21 

77 

Be 

93 
122 
1B6^ 
4SB 
475 
596 
547 
575 
507 



.3 



1 
$S 
92 
124 
1S€ 
291 
20B 
22? 
333 
3$4 
361 
410 
462 
512 
582 
607 



.4 



9 
9 
4 

e 

2$ 

$1 

3S 
4B 
39 
$0 
71 
116 
232 
36? 
490 
484 
4d4 
462 
540 
582 
573 
552 
610 
606 
638 



.5 



24 
14 
?3 
?1 
136 

lei 

16$ 
132 
140 
ISl 
26? 
200 
SIS 
494 
483 
477 
538 
551 
560 
588 
!i33 
575 
574 
594 
609 



.6 



.7 



.8 



.9 



1.0 



5 


9 


?0 


29 


66 


111 


21 


111 


149 


S4 


194 


193 


i2B 


2?2 


234 


1?4 


533 


364 


146 


540 


454 


141 


505 


498 




574 


484 




601 


546 

572 



us 



D 
0 



58 
58 



848 
282.7 



1249 
416.3 



1474 
491.3 



809 
115.6 



1S48 
193.5 



3867 
429.7 



4844 

403.7 



7073 
505.2 



8615 
662.7 



701 
87.6 



3427 
685.4 



3675 
612.5 



145 
145 



H 

T 



^ fttUfttioA Avtrmgt 
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.: Figure 5-15. Histogram of Exponential 

Parameter As a Function of 
" ^ - the JMI 

Notes: 

1) Numbers within histogram bars are 
the number of cases/number of 
terminations. 

i * 2) Data Is as of December 10, 1974, 
! 3) Class. interval is ^?;:?r^OMI< IJfK- 
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Table 5-19 shows the results obtained at the other three retention 
followup times, 

TABLE 5-1 9 

EXPOHENTIAL AMLYStS; CDHBtftED RCTENTIOft'TtHE AV£ftAGES 



As of: 


JHI 


Days 
worked 


Average 
HOt of Retention 
Terminations illmeiDaYiL 


Ratio 


Sig. 


5-31-74 


>0 
SO 


24991 
3341 


68 
18 


367.51 
185.61 


1.98 


tt 


2-28-74 


>0 
£0 


20101 
2789 


62 
18 


324.21 
154.94 


2.09 




12-31-73 


>0 
SO 


16607 
2328 


51 
16 


325.63 
145.5 


2.24 





Kott: For the analyses done us1n9 the data obtained at the three.earl1er followup 
tiRiest the JHI class interval boundaries were defined differently than was 
done for the datk obtained H of Decewbtr lOt 1974. The cUss Intervkls 
were defined ks greater thkn the loMr cUss Unit tnd less th«n or equal 
to to the ypiwr limit. 



Tables 5-20 through 5-22 show the data distributions used in the 
calculations shown in Table 5-19, 

The results given above follow the pattern established in the 
truncated analysis. One sees that the effect of the JMI decreases 
as the time period increases. This is akin to the truncated analy- 
sis results where the relationship difference for the JMI for those 
. "lents who worked less than or equal to "X" days and those who 
worked more than "X" days was maxinum at about 60 days and decreased 
as "X" got laicger. 



5,4.4 Multivariate Regression Analysis 

In order to further delineate the relative long-term effects of 
the CAI and JMI on job-retention time* the following multiple 
regression analyses were conducted using the final set of retention 
data obtained as of December 10» 1974, 

1) Using only the 71 cases where the tennination time was 
known (people had terminated) and the people involved 
had taken both SICLs, 

2) Using all 110 cases where both SICLs were taken with those 
"still working" assigned a random future termination time 
based on an estimated exponential tennination time delay 
function. 
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TAP.c 5-20. EXPONENTIAL ANALYSIS: OBSERVED RETENTION TIME 
(IN CALENDAR DAYS VERSUS JOB MATCH INDEX) 
AS OF MAY 31. 1974 

JOB MATCH INDEX 

CATEGflRy VALUE !»AM6e NO. OF NO, SUM OF DAYS 



NO. 


• GT. 




• tE. 


PEOPLE 


TFRMINATEO 


ehployeo 


I 


• 9 


TO 


1.0 


0 


0 


0 


?. 


•a 


TO 


• 9 


1 


1 


145 


3 


.7 


TO 


• a 


8 


5 


1780 




• 6 


TO 


.7 


13 • 


5 


3269 

*ff V ^ 


5 


• S 


TO 


• 6 


7 


7 


571 


6 


• 4 


TO 


• 5 


24 


1 2 


^di 1 


7 


• 3 


TO 


• 4 


25 


13 


4964 


6 


• 2 


TO 


• 3 


. 17 


1 0 




9 


• 1 


TO 


• Z 


13 


7 


2649 


10 


• 0 


TO 


• 1 


10 


0 


1523 


n 


-.1 


TO 


• 0 


9 


8 


643 


12 


-.2 


TO 








1122 


13 


"•3 


TO 


-.2 


5 


3 


863 






TO 


-.5 


4 


3 


6SS 


15 


-.5 


TO 


-.4 


1 


1 


58 


16 


-.6 


TO 


-.5 


0 


0 


0- 


1 7 


-•7 


TO 


"•6 


1 


I 


0 


10 


-.8 


TO 


-.7 


0 


0 


0 


)9 


-.9 


TO 


-.8 


0 


0 


0 


?o 


-1*0 


TO 


-.9 


0 


0 


0 










142 


86 






T 




D 


D/T 







JMI >0 68 24991 367.51 c - i or 44 

Jhl $0 IB 334T 18531 L^JM** 
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TABLE 5-21. EXPONENTIAL mYSIS: OBSERVED RETENTION TIME 
{IN CALENDAR DAYS VERSUS JOB MATCH INDEX) 
AS OF FEBRUARY 2B. 1974 



JOB MATCH INDEX 



I 


.GT. .LE. 
•9 TO l.b 


MO. OP 
PEOPLE 

* 


NO* 
TERHINATEO 

0 


SUM or OA^ 

EMPLOYED 
0 


z 


• B 


TO 




1 


1 


145 


3 


• 7 


TO 


«B 


B 


5 


1504 


4 


* 

• 6 


TO 


.7 


13 


4 


2465 


5 


• S 


TO 


• 6 


7 


7 


573 


6 






• 5 


24 


12 


. 4707 


7 


• 3 


TO 




25 


13 


3860 


B 




TO 




17 


6 


3426 


9 


« 1 


TO 


• 3 


13 


7 


2097 


to 


• 0 


TO 


• I 


10 


7 


1324 


11 


" • 1 


TO 




9 


6 


551 


la 


"•C 


TO 


"•I 


4 


2 


93B 


13 


" • 3 


TO 


-•2 


5 


3 


679 


1* 


" • *? 


f V 




4 


3 


563 


15 


" • ^ 


TO 

f w 




I 


1 


5B 


16 


-•6 


TO 


-•5 


0 


0 


0 


17 


"•7 


TO 




I 


1 


0 


18 


-•9 


TO 


f 


0 


0 


0 


19 


-.9 


TO 


-•8 


0 


0 


0 




r 


TO 


-•9 

D 


0 

O/T 


0 

"80 


0 


OMI > 0 


62 




20101 


324,21 


- F = 2.09 


++ 


JKI £ 0 


IB 




27B9 


154.94 
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TABLE 5-22. EXPONENTIAL ANALYSIS: OBSERVED RETENTION TIME 





JOB MftTCH INDEX 








CATEGORY 
NO. 


VALUE 

.fit 
• III • 


(tANGE 


NO. or 

J^EOPtE 


Nil) 

TERHINATEO 


ur HATS 
EM^LOVCO 


1 


• ▼ TV 




0 


0 


0 


2 




A 


1 


1 


145 


3 


• f 1^ 




8 


2 


12TT 


4 


. it ifn 
• Q TU 




13 


3 


1923 


5 


• ^ vn 




T 




S46 


6 


• TU 


• 5 


24 


9 


3964 


7 


. ^ TO 




25 


11 


3130 


3 




• 3 


IT 


5 


2732 


9 


. t TO 


A 

• 2 


13 


T 


1743 


10 


• 0 TO 


• 1 


10 


T 


1147 


11 


_ 1 MA 

••1 TO 


• 0 


9 


8 


492 




TO 


-•1 


4 


1 


763 


13 


••3 TO 




5 


3 


561 


1* 


M 4 TO 




4 


a 


454 


15 


-•5 TO 




1 


i 


58 


16 


^•6 TO 


-.5 


0 


0 


0 


IT 


Tn 


"•6 


1 

• 


1 


* 

0 


18 


-.8 TO 




0 


0 


A 

V 




••9 TO 


••8 


0 




0 




1*0 TO 

T 


-.9 

D 


0 

D/T 


0 


0 

r 


~M ISO " 


51 


16607 


325"76r 


■ F ^ 2.24 H 




JMI S 0 


16 


2328 


145.5 
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The analyses of variance resulting from the above regressions are 
presented as Table 5^23, 

For these regressions* the variables were defined as follows: 
Y = {Days Worked)^^^ 
» CAI 
= JMI 

The square root transformation for the dependent variable Days Worketl 
was applied in order to more nearly normalize the skewness for this 
parameter, A logarithmic transfunction would probably have been 
even better but was precluded since one of the observed Y's was "0" 
days. The CAI and JMI were both used "as is" since they don't approach 
-1 or +1 very closely and were already sufficiently close to being 
normally distributed for analysis purposes. 

In the first analysis, neither of the indices is significantly 
related to retention time. The reason for this would appear to be 
that the effects of JMI on early termination {as noted in the earlier 
t-test analyses) has been diluted by the addition of the long-tem 
termination data; while the "long-term" effect of the CAI cannot be 
seen since a considerable number of those with very high CAI are in 
the "still-working" category, and hence were not included in this 
particular analysis. 

It was not considered to be statistically appropriate to include the 
"still-working" times with the "termination" times in a composite 
regression analysis due to the fact that the resulting distribution 
of days worked is being bounded on the upper end by the date when 
w did our lest foUowup, (The frequency histogram shown in Figure 
5-6 shows the U-shape distribution one would have, and hence would 
not be very reasonably represented by a normal distribution,) 

In an attecnpt to overcome this problem, use was made of the exponen- 
tial life-testing model v^ich was discussed previously. Since the 
Job-retention times seem to behave more or less in the same way as 
a life-testing decay, an exponential time decay function was fit 
to all 94 cases who had terminated. 
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TABLE 5-23 . ANALYSES OF VARIANCE FOR RETENTIOM TIME MULTIPLE 
REGRESSIONS 

Y = (days worked)^^^ 
= CAI 

Xg = JMI 



71) 



SOURCE 
^ [CAI] 


Degrees of 
Freedom 
df 
1 


Sum of 
Squares 
ss 

1.48 


Mean 
Square 
m 

1.48' 


1 


SIG* 


[JMI] 


1 


24.89 


24.89 


1 




Resid 68 


2408.19 


35.41 






Total 70 


2434.56 









MULTIPLE CORRELATION COEFFICIENT IR ^ 0.1041 



(2) Using I ALL DATA] (N = 110) 

Note: "Still-Working" Terminations Randomly Assigned Calendar Days 
Employed To Termination From Exponential Time Decay Function 

SIG.* 



SOURCE 


df 




ms 


1 


Xi [CAI] 


1 


1224.28 


1224.28 


9.94 


Xg [JMI] 


1 


357.79 


357.79 


2.91 


Resid 


107 


13178.54 


123.16 




Total 


109 


14760.61 







MULTIPLE CORRELATION COEFFICICNT 



R = 0.327 



++ 



♦Significance Code: 

- = Source of variance NOT significant 
+ ~. source of variance SIGNIFICANT 0 95X Confidence Level 
. source of variance SIGNIFICANT @ 99% Confidence Level 



11-228 



The basic formulation of the use of the exponential distribution 
for the "liJFe-testing" application is largely due to the work of 
B. Epstein.' The referenced paper presemts the derivation of all 
of the fonnulas v^ich were employed in the present application. 

The basic exponential distribution is given by the following proba- 
bility density function: 



f(t) - -I 



•t 

e 



where 



t > 0 
< 0 



This distribution has only one basic parameter* 9 , which is termed 
the "mean life" and is sometimes also termed the "mean time between 
failure" {abbreviated KTBF). This parameter is estimated from the 
data by observing, in the present case* the total number of dqys 
worked by aU. clients {both terminated and still working) and divid- 
ing this by the total nuntber of tenninations observed. 

Thus, we obtain: 



Total Working Days {AH Clients) 
Tenninations 



38«333 
94 



407.8 days/ 

termination! 



{See the data presented in Section 5.4.3.) 

Using this estimate, it is now possible to estimate the cumulative 
exponential decay function using the basic formula. Thus, the proba- 
bility of a random client being employed for time t* can be presented 
as follows: 



P 



t* 



= e 



^ =e 



t* 



The first column of Table 5-24 presents the observed times of all 
clients who were still working as of December 10» 1974. Using the 
above formula for one obtains the estimated probability that 



^B. Epstein, "Estimation from Life Test Data," TeohnometHoe * 
November 1960, pp. 447-54. 
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TABLE 5-24 . ESTIKATIOH OF RANDOM EXPONENTIAL CLIENT TERMINATION TIMES 
FOR THOSE "STILL WORKING" AS OF 12-10-74. 



t* 

(m 

Da ys ) 

582 
* 596 
573 
582 
533 
589 
475 
596 
547 
575 
607 
462 
512 
582 
607 
490 
484 
484 
462 
540 
582 
573 
552 
610 
606 
638 

















Pi* 


R 


t 


t* 




R 


t 




















Days; 










.240 


.8739 


637 


494 


.298 


.8962 


539 


.232 


.3123 


1071 


483 


.306 


.8146 


567 


.245 


.4442 


904 


477 


.310 


.1605 


1223 


.240 


.9976 


583 


538 


.267 


.4539 


860 


.271 


.0963 


1487 


551 


,259 


.5926 


764 


.236 


.3830 


980 


560 


.253 


.3698 


966 


.312 


.0392 


1796 


* 588 


.236 


.3393 


1029 


.232 


.8532 


661 


533 


.271 


.1691 


1258 


.261 


.2118 


1180 


575 


.244 


.6142 


774 


.244 


.2253 


1183 


574 


.245 


.9804 


582 


.226 


.1285 


1444 


594 


.233 


.3321 


1044 


.322 


.1041 


1385 


609 


.225 


.2611 


1157 


.285 


.6425 


692 


533 


.271 


.6043 


738 


.240 


.1120 


1475 


,540 


.266 


.8992 


584 


.226 


.2598 


1157 


* 505 


.290 


.1825 


1199 


, 301 


.5643 


inn 

709 


574 


.245 


.4967 


oca 


.305 


.5967 


695 


* 601 


.229 


.0043 


2823 


.305 


.4237 


834 


* 454 


.328 


.0180 


2092 


.322 


.1945 


1130 


498 


.295 


.4144 


857 


.266 


.1155 


1420 


* 484 


.305 


.8555 


548 


.240 


.5028 


862 


546 


.262 


.1842 


1236 


.245 


.9603 


590 


572 


.246 


.8267 


650 


.258 


.6236 


745 










.224 


.0187 


2233 










.226 


.7708 


71? 










.209 


.6629 


806 


* Preference SI a Only 





where: 

t* = last observed time at which client "still working" (Days) 

/I 

P+* ' estimated probability of surviving time t* based on 
exponential decay e 



R 
t 



four-digit random nunter (A. Hald's Random Digit Tables) 
In 1 = estimated random client tevmination time 



R P 



't* 
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a client has been employed the length of time, t*, in a "still working" 
state. These estimated probabilities are shown as the second column 
of figures. 

The third column presents a four-digit random number taken from 
A. Hald's random digit table.^ 

The product R"^.^ will thus represent a "random" point on the total 
remaining probaoility scale for total time worked which is greater 
than t*. In order to find the time which corresponds to this random 
or "termination" point, we substitute Rp^* f or F^* in the formula, 
obtaining: t* 

^ ■ wrs 



Solving this formula for t*, which will now be denoted as t, the 
following result is obtained which represents the desired "random" 
future termination time. 



The final column of Table 5-24 presents the values of t corresponding 
to each of the individual client still-working times, t*. 

Figure 5-16 shows the frequency histogram of days worked till termina- 
tion. The crosshatched area pertains to the actual observed tv-finina- 
tions. The other area pertains to the distribution obtained from the 
above methodology. 

Using the random termination times, from Table 5-24 for each of 
the "stilNworking" cases, a second regression was conducted using 
the resulting 110 case composite groups; i.e.. the actual tenninations 
and the estimated. 

The results of the second regression were also presented in Table 5-23. 
These regressions indicate the expected result as previously seen in 
the "t"-test; i.e.. that for long-term job retention, it is the CAI 
index which is statistically significant. 



A. Kald, statietical Tables and Formlae, New York: John Wiley & Sons. 




2 '1 

^ I 

1^ 



Figure 5-16. 



4 



Frequency Histogram of 
Square Root of Calendar Days 
Employed Until Termination 
Using Estimat -** TErminatiwis 
for Clients Still Working 
as of Decenber 10. 1974 

Notes: 

1) Crosshatched areas are all knovm 
terminations 

2} The equivalent calendar days employed 
are shown below the square root values. 



Ierjc 




As a further measure of the effect of "Restriction" on the first 
data set which used only the job-terminated cases, an "unrestricted" 
estimate of the correlation was performed using the following correc- 
tion formula."' 




where: 



R,2 ~ the desired correlation between 2 parameters (1,2) 
based on an "unrestricted" range of parameter 1 

r■^2 = the observed correlation of parameter 1 and 2 based 
on the "restricted" range of parameter 1 

S-. = the observed "restricted" standard deviation of 
parameter 1 

S-. = the value of the "unrestricted" standard deviation 
of parameter 1 



^ J. R, Guilford, Fundamental StaHetioe in Psyohology and Eduaationt 
4th edition. New York: McGraw-Hill, c. 1965, pp. 341-45. . 
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If one takes the "restricted" correlation estimate of 0.104 and 
uses the estimated ratio (1.97) of the unrestricted standard 
deviation (based on the randCHtily extended exponential retention 
times) to the restricted standard deviation (using terminations 
only), the corrected estimate of correlation becomes: 

= 0-10^ - O.m 

,/l - (0,104)2 ^ (0.104)^ (1.97)^ 



Although this Is not as large as the actual computed correlation 
using the estimated extended times, it does represent about a 
factor of two times the original correlation observed using termi- 
nations only. The reason it underestimates the actual is probably 
due to the fact that the data points absent from the restricted 
case are heavily weighted with high values of the CAI. 



360 
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5.4.5 



Termination Reasons: Data and Analysis 



The retention analyses presented so far have all been done Independent 
of the reason for termination. As discussed earlier* OUrasystans 
obtained the reason for termination from the employer. The reasons 
given by the employer and the number of terminations associated with 
each reason are shown in Table 5-25. This table also shows the 
distribution of the terminations by reason against the number of 
calendar days employed. Examining the reasons for termination, 
one sees that there are many cases In which the exact reason is 
not discernible. 

As an example, one sees thai the largest category of terminations 
by reason is "Quit No Reason Given" (i.e., there were 17 termina- 
tions in this category, which is 18. U of the total of 94 terminations). 
In addition, there were two (2) terminations for which no reason 
was given by the employer {labelled INA in Table 5-23, line 13), 
and there were 10 terminations for whcih the reason given was either 
Voluntary Termination or Terminated By Company-No Reason Given. Thus, 
there was a total of 29 terminations {30.8% of the to».al terminations) 
for which the "real" reason is not clear. In other words, the reason 
for these terminations could have been the same as another reason 
listed, i.e., Quit-No Reason Given could have been a reason such as 
Didn't Like Job or Quit To Go To School or Quit To Get Harried, etc. 

In addition to this pif^bl^, there is of course the qMestion of the 
real validity of these reasons. As stated, these reasons were given 
by the «nployer. Since the people themselves were not contacted, 
no check on the reason is available. One should note that t^ereas 
also obtaining a reason for termination from the person would be a 
great help it is, in Oltrasystems' opinion, likely that in many 
cases the reason given by the person will differ from that given by 
the employer. As an example, the employer reason Did Not Qualify 
might be contradicted by the person giving a reason such as Didn't 
Like the Job or The Company or Their Supervisor . That there would 
have been a difference in perspective between the two respondents 
is, as stated, our opinion and is obviously not , supported by any 
data we obtained. 

Aside fron these two Issues, the reasons for terminations given also 
show that there were two people who never reported for work, four 
who were terminated when the business closed^ and seven who were 



All four of these people were employed in the same type of job at 
the same company. The ccxnpany did not actually go out of business^ 
rather, they shut down operations at the physical location where the 
four people were employed. The company maintained operations at 
another plant located approximately 60 miles from the one closed down. 
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TABLE 5-25- TERMINATION REASONS AND DISTRIBUTION OF DAYS 
EMPLOYEO BY SUCH REASON 



Une Ultra Total Calendar Days Employed 
Number Termination Reason Code Nos. Term, 31- 61- 91- 121- 200- 300- 400- 500- 
0-30 60 90 120 199 299 399 499 599 
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given* would be on the results obtained from the truncated analysis 
presented earlier (In Section 5.4.2). The following two sets of 
temrinatlons were excluded: 

Group 1: Truncated Analysis Excluding Tennlnatlons for the 
Following Reasons: 



, Termination Reason (Ultra Number of 

Line Number Code Number in Parentheses) Terminations 

9 Never Reported For Work (1 ) 2 

10 Business Closing (^) 4 

11 Laid-Off-Uck-Of-Work (4) 6 

12 Canpany Financial Problans (27) 1 

13 INA (22) 2 

14 Temporary Hire (20) _4 

Total 19 



Group 2: Truncated Analysis Excluding Terminations for the 
Following Reasons: 



1 . Termination Reason (Ultra Number of 

Line NunfJjer Code Number in Parentheses) Terminations 

16 Quit To Go To School (25) 4 

17 Went Into Business (19) 1 

21 Left Area (14) 3 

22 Quit To Get Harried (10) 4 

23 Pregnancy (29) 1 

24 No Driver's License (2) 1 

25 Language Problem (31) 1 

26 Violated Probation (23) 1 

27 Falsified Birth Certificate (3) J. 

Total 17 



Line Number refers to Table 5-25. 
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Table 5-26 shows the pertinent condescrlptlve statistics for the 
Group 1 and 2 termination cases and the remainder of the termina- 
tions. Using the t-test to test for significant differences between 
the means of the measures shown, between tl« termination groups, 
yielded no significant differences (as shown in the table). There 
was, as can be seen, reasonable uniformity between the mean values 
of the CJHS indices except the Job Match index between the termina- 
tion groups. One sees that the mean Job Match Index for the Group 1 
terminations was much lower than for the Case 2 or All Other termi- 
nations (All Other irteans all terminations excluding those contained 
In Group 1 or 2). The data shown In Table 5-26 masks the distribu- 
tion over the Indices over the calendar days days employed. As an 
example, examining the JMI Indlce between the Group 1 terminations 
and the remainder of the terminations, for those people who were 
employed S6Q calendar days yields the following: 



Cases Employed < 60 Days 



Group 1 Remainder of 

Terminations Tensi nations 

Z(OHI) -.1265 .222 

s .3165 .322 

n 8 30 



t « -2.73 
sig ++ 

(Note: In the above* the remainder of the 
terminations includes the Group 2 
terminations.) 



One should bear In mind that the two people who never reported to 
work are unique esses In the sense that the reason for termination 
explicitly determines the number of calendar days employed, 1*e., 
zero. Table 5-27 shows the truncated analysis resulting from the 
removal of these two cases. ^ Comparing the results given In 
Table 5-27 with those presented earlier In Table 5-16, one sees 



Note: These two cases Involve people who only took the prefer- 
ence SICL, Hence, the previous results shown for the client ad- 
justment and difference Indices are not affected by these two 
caiies. 
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TABLE 5-26. CONDESCRIPTIVE STATISTICS FOR CLIENT 
AND JOB MATCH INDICES FOR TERMINATIONS 
GROUPED ACCORDING TO REASON FOR 
TERMINATION. 



Z(CAI ) Z(CDI ) Z(JMI ) Z(JDI ) Days 



Group 1 Term 

X .382 41.81 .192 60.83 8.66 

s .469 12.12 .411 13.66 5.74 

n 14 U 19 19 19 

Group 2 Term 

X .322 48.58 .357 60.51 11.26 

S .524 14.33 .339 12.93 4.85 

n 12 12 17 17 17 

All Other Terminations 

X .368 47.29 .313 58.43 9.70 

s .0945 15.60 .341 14.70 5.92 

n 45 45 58 58 58 

Total Terminations 

X .363 46.47 .296 59.29 9.772 

s .300 14.62 .356 14.09 5.706 

n 71 71 94 94 94 

Group 1 and All Others 

t .192 -1.20 -1.27 .628 -.669 

Group 2 and All Others 

t .568 .258 .468 .526 .992 
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TA8LE 5-27. ANALYSIS OF CLEFF SYSTEM SCORES VERSUS 
LENGTH OF TIME WORKED {TIME WORK 
CATEGORIZED IN TWO DISCRETE INCREMENTS) 
EXCLUDING TWO TERMINATIONS WHO NEVER 
REPORTED FOR WORK 





3/ IMT \ 




0-30 X 


.203 


59.30 


X 


.305 


13.03 


n 


26 


26 


>30 X 


.378 


58.34 


s 


.317 


13.03 


n 


114 


114 


t 


2.56 


-.34 


sig 


++ 




0-60 X 


.183 


61.21 


s 


.316 


15.02 


n 


36 


36 


>60 7 


.402 


57.78 


s 


.314 


12.33 


n 


104 


104 


t 


3.60 


-1.36 


sig 
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TA8LE 5-28. ANALYSIS OF CLEFF SYSTEM SCORES 
VERSUS TIME WORKED EXCLUDING ALL 
GROUP 1 TERMINATIONS 

(Note: Data Is for 123 cases* of which 96 took 
both SICLs) 







Z{CAI) 


Z{CDI) 


Z{JMI) 


Z{JDI) 


0-30 


X 


.299 


49.13 


.244 


59.92 




s 


.435 


14.45 


.314 


13.68 




n 


19 


19 


21 


21 


>30 


X 


.531 


41.39, 


.379 


58.15 




s 


.515 


14.72 


.323 


13.27 




n 


77 


77 


1P2 


^ AA 

102 




t 


1.80 


-2.06 


1.75 


-.56 




sig 


+ 


+ 


+ 


+ 


0-60 


X 


.294 


48.30 


.222 


61.10 




s 


.460 


13.99 


.322 


15.16- 




n 


26 


26 


A A 

30 


A A 

30 


>60 


X 


.556 


40.93 


.399 


57.59 




s 


.508 


14.85 


.314 


12.61 




n 


70 


70 


93 


93 




t 


2.30 


-2.19 


2.66 


-1.26 




sig 


+ 


+ 


++ 




0-90 


X 


.416 


45.35 


.242 


61.04 




s 


.532 


15.08. 


.318. 


14.76 




n 


31 


31 


38 


38 


>90 


X 


.518 


41.77 


.407 


57.29 




s 


.495 


14.80 


.316 


12.51 




n 


65 


65 


85 


85 




t 


.92 


-1.10 


2.67 


-1.45 




sig 






++ 
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TABLE 5-28 (continued) 







2{CAI) 


2(C0I) 


2{JMI) 


2{JDI) 


0-120 


X 

s - _ 


.396 


45.42 

.-_J5,.4a. 


.253 
...... _- .321 . 


60.41 

Jl 4.^55 




n 


34 


34 


43 


43 


>120 


X 

s 
n 


.534 
.475 
62 


41.56 
14.55 
62 


.408 
.31S 
80 


57.40 
12.52 
80 




t 

sig 


1.28 


-1.22 


2.49 
++ 


-1.20 


0-199 


X 

s 
n 


.355 
• 140 
43 


46.95 
15.84 
43 


.336 
.344 
58 


57.83 
14.31 
58 


>199 


s 
n 


' .590 
.470 
53 


39.66 
13.40 
53 


.374 
.306 
65 


59.00 
12.41 
65 




t 

sig 


3.17 
+++ 


-2.44 
++ 


0.65 


0.48 


All 
Term. 


X 

s 
n 


.358 
.247 
57 


47.62 
15.05 
57 


.323 
.339 
75 


58.90 
14.26 
75 


Still 
Working 


X 

s 
n 


.670 
.458 
39 


36.15 
11.62 
39 


.407 
.295 
48 


57.75 
11.75 
48 




t 

sig 


4.32 
+++ 


-4.01 
+++ 


1.41 


-.46 



3()0 
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TABLE 5-29. ANALYSIS OF CLEFF SYSTEM SCORES VERSUS 
LENGTH OF TIME WORKED EXCLUDING GROUP 1 
AND 2 TERMINATIONS 

{Note: Data is for 106 cases, of which 94 took both 
SICLs.) 



iJob-Retention 



Time in Days 


2(CAI) 


z(cDn 


Z(JMI) 


Z(JDI) 


A) 0*30 


X 


.350 


47. 9B. 


.262 


* 59.19 




s 


.385 


13.95 


.324 


13.97 




n 


18 


18 


19 


19 


>30 


y 

A 


.551 


40.52 


.376 


57.89 




s 


.525 


14.86 


.321 


13.33 




n 


66 


66 


87 


87 




t 


1 .52 


' -1 .91 


1.39 


-.38 




sic 




+ 






B) 0-60 


X 


.355 


46.65 


.245 


59.16 




s 


,435 


14.02 


,324 


15.29 




n 


22 


22 


24 


24 


>60 


X 


.563 


40.50 


.388 


57.82 




s 


.517 


14.99 


.318 


12.87 




n 


62 


62 


82 


82 




t 


1.68 


-1.68 


1.93 


-.43 




sig 


+ 


+ 


+ 




C) 0-90 


X 


.472 


43.88 


.260 


59.43 




s 


.518 


15.17 


.322 


15.01 




n 


26 


26 


31 


31 


>90 


X 


.524 


41.32 


.395 


57.58 




s 


.500 


14.85 


.318 


12.73 




n 


58 


58 


75 


75 




t 


0.43 


-.72 


1.99 


-.64 




sig 






+ 





361 

11-245 



TABLE 5-29 (continued) 



Job Retention 



Time in Oays 


KCM) 


2{CDI) 


Z(JMI) 


2(001) 


0) 0-120 


X 


.416 


44.94 


.282 


58.74 




s 


.540 


15.21 


.327 


1 A AA 

14.90 




n 


28 


28 


35 


35 


>120 


X 


.554 


40.70 


.392 


57.81 




s 




14.68 


.31 7 


12.68 




n 


56 


56 


71 


71 




t 


1.19 


^1 .2J 


I .0/ 






sig 






T 




E) 0-199 


X 


.356 


47.17 


.33* 


57.34 




s 


.2/4 


Id. l9 


lie 

.335 


1 A >l 7 

14.4/ 




n 




^A 




41) 


^ 1 A A 

>199 


X 


.612 


38.68 


.379 


58.70 




s 




13. Uo 


.315 


IZ.DZ 




n 






fil 


u 1 




t 


2.88 


-2.65 


.87 


.52 




sig 


++ 


++ 






All 


X 


.368 


47.29 


.313 


58.43 


Term. 


s 


.094 


15.60 


.341 


14.70 




n 


45 


45 


58 


58 


Still 


X 


.670 


36.15 


.407 


57.75 


Working 


s 


.458 


11.62 


.235 


n.75 




n 


39 


39 


48 


48 




t 


4.33 


-3.66 


1.50 


-.26 




sig 


+++ 


+++ 
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"Job/Man Matching In the 70's," Samuel H. Cleff 
and Robert M, Hecht, Datamation t February 1, 1971 
(Volume 17, Number 3), Technical Publishing Company, 
Barrington, Illinois. 
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It's time to change those 
personnel selection techniques 
we developed in the 'SOs 



Job/Man Matching 



Tlte t\vo-aiid*aOiiiff decades since World 

MWitr II Iiflve seen the Rrcutest tcchnolopf- 
ciil m-olnlion jn mnn*s liislory. Yel (he 
ideas aiid lcchni(|ucs applied lo indastry*s 
tiiii^c lurgeit i}ral}lcm* tlic recruiting, screenings and 
sclcttiini oF mnnpower^dcveloped in the *20s and 
lirou^Ut t<i Frtittion in the Depression oF the ^30s, w 
still with as in the 7Qs. 

Tlic tremendous lalwr pooLof the Depression dc* 
LWtk permitted inunnj^cmeiit the luxury of hiring one 
ill twnty-tlie "most qiinliBed*** perhaps in reality the 
nnist **over-quaIiRcd.'* Still fn widespread use, the 
tccltiii<tiic:E which .stilted the }'eai:s of Virge him pools 
are mnlftdaptive in these times oF full employment 
wluii n 5^ uuemp1fi>incut mtc is coiisidered a dire 
sigu nF recession, The expcmlve symptom of this 
uuhulnptive set oF employment practices^ tlie high 
Itvcl oF labor turnover among fill classes of \votker> 



The cost* . . is probably 
billions of dollars 
annually. 

tint [Kirtitiilnrly nmnu}( tnc scm,* and low^skiiicfl, 
lx)tli white and htnc t^dhir. It is estimated tliat today 
wankers luwicr ^ change jahs tui the avc.ugc uF once 
iu iai<**atu)*a^Imlf yt'ars^ while tlmsc over 35 stay oa 
rlioiV (oils only tilxutt twiccus IrniK, 

hnriug the |iartitiiliirly tight lalxir niurkct of the 
K<n'<'iUi \\*nr jears, wimi cmll!^^>^ncllt at a Ccucr^I 
l*;((i*tnV CfK pliuit in Kclioucctndy Siiaml alio\*c 
t)IK^ a nianug<fiiout sludy of ciuptoyif; tuntnver re* 
tlut in onirr U\ lill tlir '1nxt 5,tKK) johs' on thu 
tM>rnll I7,5(H> to 2?i,tMM) jHMipIo were liiK»d fn mic 
jtMvt Aiul tills omirnnl a ticaidc I>cf4>rr the "dlseov* 



ery** of tlie hard-core uuemptoyed. In today's bbor 
market, empbyer aFter employer has revealed that 
v^rtualIy every business hns its own trigger poind that 
when employment passes a certain level nn annual 
ratio of l>etween 3.5 and 9*0 lures per job is not 
unu&uuL In one situation^ an employer with whom we 
worked Ims a Job category fn which he experiences 
over tOO^E turnover per year. The cost for die entire 
economy-in rccniiting^ (niining, rework, scraps union 
grfevMnceSt lower ptx>ductivityt tiimed<plf customers^ 
k>w monlcHs probably billkms of dollars nnnually. 

The problem is just as severe for the job^seeker. 
Tlint people frequently seek and lind bw^ ^tixl semi* 
skilled jol}5 on a bit or mks, trial and ermr l>asfs hat 
long been known. In simpler times it was possible for 
a job-^eekcr to have some knowledge of tite ciniteut of 
many of tlic jol}S he was likely to get-in these times 
be is fortunate if he knows the reat re<|uirements of 
more than a hnndful of jobs. So he searclws by trial 
and error. And liecause it is relatively easy to get 
niKidicr fob at a1>out the :inme tiay levels he tries and 
errs and mom* on to nuother trid and forth. 

Tliat maimgemcnt often tends to lUrc on the same 
hasiSt with litde or no attention given to the work* 
content prcfercuce^f of the people at low* and seml^ 
skilled levels is-by dcfiuition-c(inafly tnicj titese 
peoptc do not hire tlicmsclvcs. Tlic HIack Revolution 
of tlm *(K)s was as much a rcndt against the Irrel* 
evaucc oF the employment practices of industry as 
ngainst any other single factor. 

Tltc NatMUial Assodntioii of Mnnufactnrcrs was 
^ftniKgling with the aft-iuMird t>omt>laitits of their 
mcnitwrs atiotit tho imt of turnover At the muuc time 
thiit it turning its Mtdal ciuiccrn tn tltc pwihlom of 
hard-core nucinptoymcut, nam, aware that It was 
dealing with Iwu .sides af the K;imc conn t<Kik tii>mi 
itself tin* rcsiKnislhility to dovclop n jsohitiaa. In 1006, 
Ridmrd Conardlc and Wright lillint lt^<l tlii!& effort 
thnnigli the ('enter For tndci>eudcut Actitai. Michael 
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in the '70s 



Youclmli, computer atul sy^items coitsuUaitt to l>ottt 
KAM md to t)ie New Jmey Department of Labor and 
hidustr>\ Wits as^iigned \)y both groups to find ^ wny 
to homejcs t)te computer to t)ie solutibit of the problem 
of mntebiiifi t>^pl^ itnd jobs. The na^i* ;^\ Youehab's 
suggestiotu took the sufest and wisest of research aud 



... a revolt against 
the irrelevance of 
the employment 
practices of industry . . . 

devebpnifiit ;ti>pFOnebes-two completely difTerent 
sohitjoii (Ifrectiom at tlie same time aitd in parallel 
The lirst was a computen^tkin of tlie traditional 
extniwic {oh bl>cfe» experience, education, etc., ap* 
proaeh; llir second wns *Vay out,** serappinf; all but 
tlie httrttisie oiieitcc af the IntdilioiutX ttsing tlw 
comtmter as a nuK'hine tool* 

The Misifile and Space Division was asked to 
propose an r^l uutlfne for tl^ nontnidilional ap* 
proacK Under the direction i>f Louis ClmitK), tite 
divistoi) pnt ti>j;etlu>r a conceptual appmaeh and con* 
vertcil it Mo a i>«iik>sal to kam, Tlius^ in llie spring 
of iOQG, titers wliat was to l>c u mmi xi^jn\&* 
cant and :tut*oessfnt rcsearcb i>mgr«m into tlie man* 
power xe)c('thin process^ CQmt>leteIy stipiH)rtc<l \yy pci* 
vate, ninif*^>V(>nnucnt finuUiiK^ i>nivided liy kam, Lilly 
FoundatUnt mul TIte Steru hVmily Fuod* 

Tlie ohjctlivt* of the rcsc;ireii was to dctermhie 
whctlmr and Im' jolxs nnd ft>t> j)pplle*nili conM I>c 
matdnid ttt the sciui* wtul Uiw*\UIJ«l Uwek in Itoth 
white uinl bhic udlar jol^s^n assure a higlk'r de^r^ of 
job sutisfartiun fur succc*>ssful atiplicatits und» ^mul* 
fcnicDnvb*, hJ^ilwr pUHhutivity aiul kiwer vasix htv 



by S&mue) H, Cleff and Robort M, Hecht 
employers, 

Underlying the research wits a \mk\ common st%m^ 
principle, long part of uur iiulnstrial fulklore tiut 
never venfied jEctentificatly and ctuuistcirdy ignoreil 
in practice by Iwtb iwoiile seeking jolw and empk>>'eis 
witli johs to fill. 

The pnueijple is tliis; 

fn toohiig far work (tis clscwhcrv iit life), i}coplv 
teek out those <iciicUic$ in which they feci ihaj eve 
more likely io Ite sitcccnful Cotwerjfcly, ihey at-oirJ 
those acticUics hi which they feel h*Hx likvly io bv 
iuccestful Oecuimtionally well-cilfmictt j*c(*;)/c*^ 
those ivho like tvhat ihey tla and hellvve ihcy (in' 
doing it woll-are $ignificotiily more likely io Jp a 
beiter iab for iltclr emitloyers^ aud ia siay on ihviv 
fobs longer, iho^t ;K*o;>/r n;;io orv ovaiinitianalbj mttU 
adiusted-ihosc who tk> mt/ like whai ihnj <h aud 
believe ihey are noi r/oin^ tcelL 

A lirofld cross sccltou of 20 natiotuti and ttKi\1 
emphiyens in tiu* bulianitpolis area prodded ;i \%irict>' 
uf luitt to be tilled and real life fob sittiations fi>r 
analysis. Seven Indianapolis cmplovcrsi and tlic i:m* 
DiViskni of Honey\wll, lne.» in Brifiiittin, Maw,, viH>i> 
erated ftilty fn a sop;tmte study (Project Indianap* 
oil's} to vididutc htilial fiiuling*!. 

Ill qU, tlH! research nnd development pmfjnnn- 
formulation of )i>'putheses, expcnmcntatioiL iitatisticiU 
analyses, verificatinn-Jasted Jtnme two-aiid^^ttic^lmlf 
years and pnHlticetl two mu{r>r rc;aiUs, 

Firsts a universal "UuiJimge** or umimon set of 
vuriables, wa^ hwc:ntc<l fur di^icribing in iileuiiml 
tcm!t tlie actual netivitlcji content nf tiuy scini* tii Inw* 
skilled job h> dte oammny-mul the ttctuiil Hctivitics 
prcv^msly pcrf;irtncd and currently t>it*ferred by iH^o* 
pie wIki jteck to (III that ^>l> {or wl)o (ill it ih>w}, 

SccoihI^ tlic l»i»ic prhiciple un(brlyin(( tlio ro(*arcli 

1, Th* Indtani National BanKt L. S, Avrtt ami Compinyt GoU 
ttit Lit* InauiaiKo Co, of American G«n«ial Mofqri Cqtp, 
tATtlMO Dtvt»ton)t imMUA aon T«itphona Cat RCAi aivd weiOira 
Sittctrte* 
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\v;i\ r<iiiii(l \MlifL Omt])titi<iii>illy wclkulfiivtod ]Kxi]ile 
\vl»jtt(! prcfcrfiia'jf md UiMory niiuHi 

iittivitU^v rrtiiifmiiciits tlo kiiO to .stny on tlicir jot>s 
longer ;iii(t tlo trml tfi lie ftlciitilicft comktciitly Uy 
llK'ir viijwn'fMir!! as *'!ioa'CX\foi;* Socli fndivitlonlx can 
now Ik* klciiUlicd acctiratcly in advance tltroii^h use 
of job applicant profiles made possible by the uni- 
versal lanf^nage. 

Miilc or femiile, blaek or white, urban or rural, high 
school gnidnnte or elementary sehoo] drop out, hai^* 
core unemployed nr steadily employed^N can be 
dexerfbed In temiN of both an activities experience 
profile and an activities preference profile. Both pro* 
nlt^s can l>c compnter-matcbed against either w)tite 
colbr or blue collar job profiles written in the same 
language* 

Furtlter, this language permits mathematical com* 
parisnn not only of job mid job applicant, but of job 
and job (within and l)etween companies), and appli- 
cant and applicant. 

t fdlkiws is a brief technical description of how 
the ClefT Job Mati.bing System was developed and 
h(iw it can l)e applied to the mutual benefit of em- 
p]o\iers with jobx tn Oil and the applicants seeking 
jol).\. Deuiilefl statistical revicw-^fof all reported find* 
in^v arc at.iilatite iu a technical paper. 

Neither the cniptjoyer w1)o has f^one through 1ialf*a* 
dozen Inres to filt a itingle job sati Jaetorily nor tlie five 
mL\pl;iced prcvicms hires need to ])o convinced th;it 
something wnmg with the system. In fad, a lot is 
\\*nin^ with it. 

Willie Project Indiiniapolis \vi\s in the early 
{lixcib.\inn xtagc, certain Ui^ic facts detnandef] recog- 
nition: 

K Not cnily was avnilnhle resctirch xctnity, Imt no 
.\ncce.\\fnl atlompt ]nid c\*er ]>ccn made to build a 
\yitcm f if ^irt^mi/in^ cont-ept^ wJdtb treated the field 
of himKiii rcsunrcc.\ mauii^tment in a system;)t{e way, 
with oxpcriinrntalty testable \o1utions to pn)l)lems 
fami thiily by cmploycrv, eni])]o\'ees and job seekers. 

2. Jo,itit1c\, wbitb now tutal 42,500 li\ted m the 
U.S. Km])k)>'nieiit Scr\*itCA* Dicihmnj n/ Occci/w* 
iiim<xl TUh\t have more and more 1>ect)ine social 
kibcis, and lcv.v and 1rs.\ nscfnl in dest rihin}} the job 
jt.\clf. Tlni.\, ;i 'linilding engineer* mops down the 
li.dh .nid a **firnnan" on a cticscl locomotive does— 
wlmt??? 

1*01 thcM? and R*hitcd rea^uu, it wa.\ agreed at the 
onl\;i that ^tanlla^d hiring and Job scekinj^ metWds, 
with their rc*Uancv on hujic, binich and tratlnunn had 
lH*come iiicrtM.\in);{1y oliMilctc. 11ie rcsourcli ond de* 
vcl(»|in)oiit objectives for Pioject Indimu]Hj]is tltere* 
fore wi'ii'tkiwn ;\x follow^fi 

1. 1o in\X\ti^ute an atialytictil uppniach and de. 
\ cli^p the b;i\ie methmlolo^v for matching pco])le and 
jobv at tlic law* anil Keiui-\killcd levels. 

2. To viiUdatc tlu*xe conc<*])ts niid mctlunk and to 
drvilap a nifK-liani/iihlc o^K^ratinnal model for nnitch* 
in^ ]K>ople ;ihd jobs ;il thrxdevcN. 

A !kyAti*in\ analyAis appnmth w;is taken toward 
nH*i tin;; {\\vh- ohjKli\t^. SLiti\tic].nu familiar with 
hitlr;ina|Kili\ wrre hired to tievetnp rt*prcNentalive 
.vinipIeMif .t^ailablt jntuaitd thnmitatl) ancinpkiycd 
i<ih Mvkm, With the In-lp nf tin* 20 mijicnitinft 
einpkiyers, a .\aniple nf 57 o|irn job ejite^oriej( w«s sc* 
le^ ti*ih *r]H*n Mieitil wnrkns fnini Manner WwtKc inter*" 
virwi'd a viiinplo of i50 ehnmieally mieinpkiyt'd t>co* 



])lc, blitk .nid \\hit(\ male and fcnijlc, wtio niifcht l>e 
ctnK^iderc*d for those jolts, 

Moth the jobs (tlmiugh inuimlwnts rtiw) super* 
vixors) and tlte jab ^H'kers were inter^-iewed indeptti, 
All inteniews wicro taped and lranseril>ctl verliutim, 

The interviews when anahzed pnKlnced an ex- 
haustive sample of 2rlfK) hnman activities or behav- 
ioral units reqoired on the jobs: anil an e\^n larger 
group of 3,500 l)elMvioral unit& from the 150 people 
interview-ed. It was tlien possible to nttempt to de- 
velop a set of dimensi^m!; commoa to both pools of 
behavioral units. 

All the behavioral units g1e.ined from the people 
interviews were initially grouped .lecording to their 
relation to one of the three biisic orientations common 
to all behavior; Things, People, and Ideas. This tHad 
is as old OS Plato and is aho used by the 



SAMPke *'0)MeNS10NS OF WORK'* 

T™ThlnS»— b«h»vlor mott imm«dUt»ly »n<l 
directly orUnUd toward 00fKr«t« 

T1. ATHCCTiC 

Job,ftct]vlt]» which lmmedl»toiy »nd cfJ* 
rectly Involve the worit«r wiUi thTnt^ tuch 
th»ti 

* * he uMft th* ttrs» mu»ci«t of hl» body 

In h»rd phytic tlUbor 
** hm p»y» tttontlcn to bmct ov«r»it 
* ratutts only 
<dii ditchttfti »hov«i co»t, ic»d trtickt) 

T7. CORRECTION 

Job cctlvltl«t which lmm«dl»t«l/ »nd d\* 
r«ctly invciv* tli« worit«r wltli thini» much 
tli»ti 

** he corr«ct« th» Qu»nty In hit own 
work or In tli« work of otli«rs 

** h« •n»ure» Qu»)lty p«rform»nc« of 
mtchlntry »nd ftqulpmtnt 
<p»t up mftchlnery, rep»ir p«rts »nd 
equtpment, Intpect work of others of 
»»tf»} 

p — p< opi« — worker bchtvlor mott lmm«dl* 
ateiy snd dirsctlv crltntcd toward 
ctli«rpeop1« »r;dMilmit» 

Exsmpttt 
P3. manaqcmcnt 

Job »ctlvltl*« which tmmedl«t«w »nd dU 
recOy invoiv* the worker with PeoPie tuch 
that! 

** he iufdee, Infiuencee, or dtrecte the 
present and/cr future onttoint behav- 
ior or othere 

** he tekee or eheret reeponttbnity for 
reeuite of thnt behevlbr, includlni 
their work 

** there le hlth potentlei for emotlonel 
reietlcnthipe 

<treln new wofkere, tupervlee work* 
ere, monitor or tuerd, hire employ 
eee) 

I — idese 4t infomietlcn— worker behevlor meet 
Immedfeteiy ort^ted 
toward Ideeft. eym* 
boi5 and Inrormetton. 

E^^emptet 

t3. CCCRICAC 

ivb ecttvUFee which Immediately end di* 
r^ctly tnvoive the worker with ideee end 
eymboitt «uch tliett 

* * he recorde or ordere date of eny kind 
** proceetee paper work intended to do 

that ^ 

tflie lettere .or carde. update atock 
recorde* keep record t of tranaactionai 

eto>) 
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Dqiiiilniriil nf I^ilxir'K lyutiututnj nf ()trii;\ Un\n\ 
TUlrs^ AIhmiI i^ti nf ihr lirlcivinrf fioin Ixilli ^^oK iif 
{liilti (rll tiiln llir ThiM^\<Hdriilcil mlr^iiry, Simic 25*if 
iif iIh* |Htitilr*iiik*rvim iiiiils iiikE iif tlic jiilu 
iiilcn'k'w iMiilv {iiiiic miller ihi* tilt*;iv iiilc^iiry, Tlir 
n*iiKiiiiiH^ 15'/ iiiitl MKf n'%|icrlivv)y, wcro ^n>ti)icO 
tinder l^^l|lK^ 

(Th iiiij-fMir fainilhir Willi llii* ii^yclKtlo^u*:!! k'^lin^ 
<liiiio in vixiilinnnl ^nftlinicc niiO iiiiluvlry ^ciicmlly, 
lliiv JisliiUiilioii \\ mnst iiUcic^liiif^ Siidi IcKliiii; 
usually is nihcctiu*d primarily willi 1iK*;is, and wliilc 
il is Kn (^iiiccnicd wilh htipic, very liUlc Icsiiii^ is 
dune in tlie TIrings :irc:i. Tliis ii\:iy liolii eKjdiiit) tlie 
Inw (trntii-li^T vjilidilics of iisycliotii^ir^il IcslinU iw 
liliic (ttlliir omid:>ymei)l as well <is tlu* rchUvcly lii^li 
vnlidilies of siicli testing for raif9;ly clerical :uk1 nd* 
miittslmlivt' ji)|>sj 

SulHMle^ory defiiiUions iiii|dicit in the dalsi were 
disc^weicd uiid defiiird fin' the lieliiivjonil iniits cid* 
leclcd ill llic p<,ti|)|c iiilcr\*iew:„ Tliri) tliesc were ii^^ed 
as tlie b^i^e fnr clissifyiim die iiiiils frnm llie jnb 
interviews. Wl>nl rcMilled wiis ;i set of 16 '^Dinieii* 
jfioiis ul \Vnik**Ml]e l;tngn^i^e coimnmi to l>ol)i men 
and jnl)K. Ei^lit of die 16 dimemioiis pertain to 
Thin^v :nid Tour each to Pciiple and Ideas (see Ik}x 
at lert). 

After moiitlis of sorting, teittmf;, checking, ainl 
statistie^d sc.ding, we developed three liaxie tools^-one 
for U!cc with jot) applicants, the nther two for n\e widi 
superviitory iimonnel— Imsed nu ttie t6 Dimensiaiis of 
Work and contaiiiitig l}eliaviOml units collected from 
people aikI Jobs, 

Developed for job seekers was a Sclf*Iiitervicw 
Clieck List, a prep^ttemcfl, self*administered inter* 
view fomi. Appliennts, unstiper\*ised, go tlirough the 
form twice in about an hour First they indicate those 
units the>* like t>e!ct and dislike most (i.e., wliat moti* 
vates diein iwsttively and negatively)^ then they in* 
dicate tlK>se dtey Itave done most and done least. 
Wlicn scored, two highly reliable applicant profiles 
result: one describing activities preference, the sec* 
ond activities experience, arranged according to die 
16 Dhnensioiis of Wovh. 

For sni>erviNnrs of jotxs to l>e (Stled, we developed a 
Job Outline ChecS: List and a Job Curd Sort: botli are 
completed In about liatf an Iwur widi the help of a 
penoniiel representative, methods man, or Industrial 
engineer, us appropriate. Tlie chet'k liift de!ceribes tlie 
specific rrnuiremcnls of die job. The card sort de* 
scribes the job in generator glnlml terms and serves as 
an fiiternul cnnsisteuey cheek on the specific job 
profile. An nccurate profile of joh r^qiitremeuts, both 
positive uiul iief^ative, re^nlts. The profile tlvcn can be 
matcbed with the profiles :>f npplicant!^ for the job. 

Fig, I itlHiws tbe pmnii*:; nf twn different joli!^. Fig, 
2 (see page 20) sliows tbe cxpcriciiee/preference 
profiles i>f sncccN^fnl applicants fnr tbeso jnH 

The two t^er^kin profile^i in Uig, 3 are eaeli excellent 
mnteties Utr a diirerent iine nf the two jnb pn)fik*s 
shown in i^ig, L Tbey are caeli classic mivmatclics for 
tlie odicr jnb. 

Wliat wai» learned Initially In Indianapolis and 
sntijtei|nnitly vatidati^l in Indianatioljif and Driglitnn 
wax tliat tlio eloNer the i>er?itni/job match intd the 
higlH'r the prcferencc.A'«iH;riiJncc cnrrelatjon die Iwt* 
ter ttte liiro^fnnn tlio vicwtxiiiu nf ctnplnyer and 
niip)o>n*nlikt*. 



by Kn\'eiiiliiT H)07, we wen* riMdy tn K*M tlu* basic 
nrganixing princititr ninlrrlying tln^(* clfni t\— nanirly, 
tliat twcujmtitHUtlhj wvH'tuliu\tr(i /^ii/Wr ic/io an- 
wvH-imtishvd iti thnr /u/iv tnul in do a hrttvr {uh for 
f/ir/r rni/i/or/i*rv ttnd to stmj oh thrir loin himn, 

lliese te^ls wcir etiniliietrd in Ixitli Indiaiiatxihs 
niid brightnn. In tiuhaiiaixdi^, thr particitiaiits m*re 
7 entplnyers nnil 177 of thdr i*inpln}*ir.\ fmin 2:) 
widely varying while cnllar and blue i^olliir jnt)^: hniiK 
teller, electvwnics as\eiiililen sides dcik, Ijuifcr iiui* 
chine npenitor, i^niipitter npcnUnr, tcleplunu* repair* 
man, and odH*rN fr^^m factciry, n*tail estublislinicnt 
and nfliee, The einplnyees had tn have twn tilings in 
common— u'latividy Iiing scrvi^i: (0 innndts tn 2(t 
years) and aver;ige t>r better nitiiifV! by dieir nian»i|fc* 
mcnts foroii*tlie<)nliitncce3^A 

In Brigliton, HoncyweHs eoi> l>ivisioii selected 1 13 
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Ftg. 1.. 

people who made up 12 job gmuixc of 7 to 13 
individunis each. Four of the group;; were reg^tnled 
by management ns unstable, eight groups were con- 
sidered stable. 

Job Outhne Clicek Lists were ciomptetcd liy super* 
visfiry personnel fnr all jnl)^ involveil at l)oth loe;i< 
tious. Self*lnttirview Clieck Lists were completed liy 
all 250 enipInyeeSr alwut n diird of whnm were a\ketl 
to complete diem again one to fnnr \wek.s later, Tliese 
lists were adminbtercd by liridly tn^iiicd i)ersinniel 
people^ in ordt*r to\imulate n|teratiinial ciniditiniis. 

In twdi Hrigbtnii oikI Indiannixdis, all ptutlcipat* 
jitf; empto)X*ev nbn umiplotc^l a datn sheet wliieh 
incliKled tlirce Nc^aled qncstHniM cinic*cmiiig jKYtx^ived 
success on the job: 

1. How well prqtarod were >^nt to your job, in 
compari^nt with yniir fellow cmphtyces? 

2. Wbat arc yonr chances fnr t>mnmtion in {f>m< 
pnrivnn clc? (Conttwunf nn pfir 2t0 
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Fl£. 2. 

♦nic! i^vcry response recorded. StntfsHcnl anfilyses were 
ihuc by computer, Tlic evidence elearly siipt)orted 
the 1i>*|K)thc^eNth;it: 

I l-nr lon^-term emplnyees ret^firded by tlieir man- 
iil!cnients as successfnl mHlie-{oK tlicre was n con- 
\\\ivni im\ verj' xij^iiificnit positive correhition ht- 
twiH*ii ilie employee's .\eir-retK)rtcd activities history 
mid tlu* mnn;it{enient-de\cril)eil j^ti requirements* 

'1. Man;ij;ement 1i;id nitcil endi employee's on^the- 
jiili MK^icss US iimiit^e or liit^lier; when employee self* 
nitini: sunilar, self profiie:^ tind jnh profiles 
lUiirohtKl niMth mnrc eUisely thnn wlwrc success rat* 

-\- At Uoncj-wcII, where no cdterin lor successful 
lyvrifminiKic wcrt* n.\ed tn ilKTereutfnte Inrtween 
\wiikrr\: xero «r<ler piTMni*tii'jiih.\ m;itdic\ in almost 
i*\vry iMse identified inrmlwra of **unstaMe*' wiirk 
^nni|i\ wlucli li£iit c<ni\f\tiiit iinil cimtfnnlii|; high 
cnipliiyir tnnuivtir, while mutches in the stnlite 
^nm\i\ wcjc Ajjinifitinitly hijiher* 

1 )lHi;diiVty iiF the pmfiU*!; t^*neriite(l 1)y the Self- 
liitrr\)rw ( hvvk Li\t iu the le^t-retest experiment 
W'AK i*\n*ptlini;il1y hit^h, with 0 ine<Uini hiterprnfilc 
nirrebtiim of k-M for .letivfties experienL*c mi +*88 
hiraitivitir\ |)i-orerc*iic€. 

An.)1y\i*s nf the Iniliiiniipnhsniiil Ilimeywll avinlls 
k-d ii\ in (iiir di-vrlo|iini*i't nf tlu* niH^nitioniil model. 
Knr e\;mipk\ wv rmnul thnt empkiyei*s whi consfct- 
rritl thi*nty*Kv\ iiveriit^c nr nlxiw nn the ii)h, inid hnd 
;i\eruut* or silxivo prt^femut* to extK*rien*v c<m*ln- 
tiiiii\, wvi^htisl jireferene<* iiml experieiUM; in ti sys- 



teni.itu*d1y iltfrerent wuy thiin tlKi.\e whii were licbw 
tiwriit^e on Ixith nK*!isiires^ \Vc nisn disiiivereU that 
the .s|H*eific reituircmcnts of tite |oh were weighted 
miuh moK* tli^in the \;ihie.\ derived froni the t}<^ienit 
dexc-ription of the joh. Our c^nnputer xy.\tcins model 
tnkes these fiietors bito aixonnt. The system uses 
tIte liotter*m£itchedt l>etter-£idjiute<t, hi|;lier<^nluuted 
wnrkcrs as n mmlcl. 

Joli priifilcs and applicant pr*^fi!es nre compnred 1;^ 
correlation fnr ^imihlrity» and h\ a difTcrence statbtle 
to control for mafor dtfTcrence^. Tl>e cx>rrelntlon com^ 
pares the shai>es of the profile^f and is partially con- 
trolled Ity the reni 'icto center nf tl>e profiles. The 
difference measure compares the extent of diver* 
gences between sets of profiles of ne;ir'*similnr sliapes, 
The computer can search a file of hundreds of jobs or 
of L^ple according to tliese analytic search strategies 
hi seccnds. 

Three important facts nre pn>vided each time an 
individual person [s matched to '*t>ank*' of profiled 
jobs by the computer^ in reni time via time-sharctl 
service; 

1. A Preference to Experience Index» 

2. A Person'to-Job Index^ and 

3. A Difference Index for each Person*to-Job com- 
parison, 

Pig. 3 (see \iage 27) shows a typical print-out» with 
these indices noted. In order to be screened into a Job 
the applicant slKmld have each index at or better than 
a companyKletermiiicd score. The jobs (ire printed out 
in mnk^rder according to either of the person-to-}ob 
fiidices. As in any referral-placeinent-seleetion sitna* 
tion, tIte efficiency of the systent increases with the 
numl>er of people and the number of Jobs w(th which 
they are matelied. The system will go either way-^it 
will match people profiles against a job file or Job 
profiles against nn applicant file. 

Typically^ a comp;iny stores 20 to 30 job categories^ 
those representing its highest turiiox^er^ and matelies 
applicant profiles against this file. Obviously it is niso 
less expenvive to store the usually ^mnller number of 
job profiles than the larger nnmber of applicants. 
Paper tape stonige for relatively inactive files {s possi* 
ble and can be loaded into the computer when neces- 

. . * correlations with four 
to five times the predictive 
power of the "normal" 
psychological test battsry . , . 

snrv'i In :i placement service sneh ns nn employment 
ageiic% private or pnhliCf it nivkcs setixe to stnre the 
profiles nf ;K-tive joli-xrckers, niiiU'hing enipkiyer re- 
i\{\csU (in the form nf job pnifile^) agahist this peo|ile 
rcgixter^ rcnuivinj? the perMn/s profile wlicn he hns 
boat xUi'Cciixfnlh* placed. 

What hiive boat imr validating e\pcrleiU'i*.f? The 
experimental v,diUiti -i .\nidy dime in hidianaixtlis 
and nrightiin devoki^HKl avenijLi* [nKiple-to-job corre- 
latkiii.\ with four to live tinuv^ the predit-tive (Kiwcr nf 
the "nonnal" p\yeliologK^il lot battery validations 
with ^im^ar work groups. We have fiilhiwctl up Jioinc 
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i)f tlic liHlinii;itK)lis .Mibjcc^^nxiiiji twiKiiid^iiOuir 
yviiTs ;ihvr orf^iittil '*mU (olli*dii;ii. In a liaiik. 
fllMlriiiin* nxnpiiiK, iiiicl tolctiliiilir (ir>m|Mliy (;i ti>t;il 
oF pIkiiiI HS nii]i}(iyiHv«} wc round tli;it lii^licn 
muU-linl cmpli)>Ti**! Iiiid Tnim 15M Ut 25f)% mcirc 
mimtlvN wiVli vUHr u>inp;mie.<t tliibi did \\k>sc wlui Imd 
Ih>)ow avcnigc mAU*tic*!. In u niu'^ytsir :dndy \\\ i\ l;ir^e 
su|>rmitirl;rt cluin, wlicrc wc Ntadicd 30 "stctitdiird" 

JANE DOE 
147^16^4906 
JOB APPLICANT 
CXPCAIENCe PAOflLE 

*6 0 0 ^*6 4 *6*1;a 13 13 t «1 3 $.3 U 
PAEFEACNCE PAOPtLE 

*l3*t3 6 O ^7 *3 -4 «7 10 to 4 1 3 7 4 5 
COMBINED PROFILE 

*10 3 ^3 ^7 1 ^10 13 11 3 0 3 6 3 5 
SUITAB(L(TY PREF* TO CXPA, (S 0.7467 

5 



« PROFILES ABOVE MINIMUM 
HOW MANY DO YOU WANT TO SB& 



3 5 33 O 11 1 



SUITABILITY INOE^ 15 0*7063 
DIFFERENCE INDEX IS 763t 
OPERATOR 
ABC COMPANY 
JOB PROFILE 

*t1 «0 10 ^3 *^ *B OOS 17 3 

6UITABILITY INDEX IS 0,6300 
DIFFERENCE INDEX 16 001. 
COUNTER CLERK 
ABC COMPANY 
JOB PROF*LE 

07 *% *3^1« O ^3 *«^15 13 *3 3 10 15 13 14 

6UITABILITY INDEX IS 0,4163 
DIFFERENCE INDEX 16 1403. 
SUMMARY CLERK 
ABC COMPANY 
JOB PROFILE 

*1t 4 10 <3 ^3 10-16 « ^6 O 
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6UITAB1UTY INDEX IS 0.3313 
DIFFERENCE INDEX IS 1460. 
AUTOMATIC OPERATOR 
ABC COMPANY 
JOB PROFILE 

<1 12!*13 3 *3 10*13 5*13 O 

6UITABIUTY INOBX IS 0.1373 
DIFFERENCE INDEX IS 1773» 
TECHNtCIAN 
A8C COMPANY 
JOB PROFILE 

<13 -A 14 ^6 <1 7 10 «5*3*t3 O 
Ftg. 3. 

long icrm successful emptoyccs and 35 "Iprd-core" 
cmplo>*cos, Wc found miicli die snme thing, Among 
llicfr stctndnrd employees^ diose with hIkwc average 
matclics were wfdi tlie ct>mpany four to Ave times 
longer than tlicue with Iwknv average mateheSt In tlie 
"biird-core" group, six^montKs tnnhWer \va,s 78^ ftjr 
t>clow«aver»igc matehes, 44^ for alKwe*aver;tg? 
matelics, In Cliarlotte, N.C*» wliere the Cleff J«h 
Matching SyMnn (cjm5) b liciiig to plac^ 
largely semifitemte lilaek fambhands in traditionally 
white indn.<ttr\\ widi a \'er\'Mn;ill jnh hank, tlie liirc tu 
refemd rate is iil)oiit 75iir"*witli rctentbn at alniut 
70% after 90 dayN. The M-Ntem \m recently Ix^gun U\ 
refer glietto dwelleni in NWarlc^ tlimngh the HiLshie^s 
& lndastH;tl Coordltnttmg Cmmeil, iittu .i juh register 
of tnily 50 jtd) categories rcprcieiithig wnnc 20()-2S0 
0|>cniitgN, 

Tbc |ires,<(ihg need for a ^vorkulile mid liolistfe^ hut 
analytleal, joh mateliing •!>'Ntom lti.<t Ix^tin deimin^ 
stmted liver ainl over *iguhi hy tlk' many fitscfnatfng 
uppUaitloiiN wlifeh otlieni Imve suggested. We are 
now experimenting wUli ittime of tliese snggcsUoiis, 
For exum|:1cj In coopenitinn with New Jersey l\e* 
Itnhilitatitm OmimNsioii, we ;irc mlng cjMs for stitne 
of its HiViits. Wo devHiitK*d a timmliire fur aiialy/.inj% 



a client**! pliyi^KMl antl eiiKitlonal eatKK-itit*N and \u* 
L';ip;inti'*\ nsing dii* .^tami' Ht l)iin(-ihf<iih nf Wmk. 
Tliis pnivi<U*.<( a var,itMmjlly fniietional piofilt* of the 
reluihilitation eUriit. A jt>h ix*giAter nnitaining miiiu* 
2St) jnh pniFiles, (lev<-li*|jc*d fcir Nr\v jmvy l**nitiltiy^ 
meiit Si*r\'i(v* t)iovideN a laige s;ini|>li* of jiAia against 
whieh the sc\ of iK*rMn^|^)tilc2! me nanpared, l(t>tli 
client and (.^tJiiiLselar tlini \va\v a >et nf teal \iih\ to 
wliieli a client may he roferred or wliicli eun M*rN"e a^f 
a realistic h\s\s for planniitg viiL'jitimial rt'ltihilitatimi. 

Working v^idi a aiiinty vueat!<inal Ncluiid &y>teni 
we arc planning ta ii.<te die sysivui to proltle indn>try 
example*! oF tr-idc^f for which ;(tmlent& aie traiix.*^!. 
This .sample l*;Iii he (.^)fn|).ir(.*(l ta the profiler of die 
courses tlieaiNelvcs hir gnidance in adjihtiitg dicm to 
iitdustry'is requirements. And, of L*tuim\ :ttiidcnt giild^ 
ance and ptacement iv an ohvicaiN spin^ulf, 

Management people \\*e've .\iKikt*n to have a>ketl ns 
to use tite sy!item for L^imparing johs am! hnilding joh 
heimrehies^ and developmei>t ladders. Odici.s hive 
suggested using it to !ieleet eadrcs fi>r new piOilnction 
lines or as an iiidn.\tilal engineering tool for t^nistrnet^ 
ing jol)S which match the proetit wirk-fonc. The h\t 
of suggestiom; goes on and on. Onr re.\pon>e U sinitdc: 
we bclicvQ tliat the system cun help da duise things, 
hut ^ve do not ^iioit> if it can, cj:^is is L-crtuinly nn 
panacea for all of our human resources prohlems, Init 
the avAilable evidence .suggests that it is \vurtliwhile 
to test it In these roles, ■ 




Or. ci«ff Is vies prwtldtnt of 

of AOP Pvreonnfil OmXm Syt^ 
tvmfti Ino, Previously, h« 
worKfttf ml th« C«nUr lor 
lncteP9nd*nt Aotton wh^rm 
M concslvsdi r«ss«rch«di 
snd da V sloped iM opmrm* 
tlon«l m«n/lob mstchlns 
sysUm» Hm rscslv«d his 
doctoral denras ?rom tH« 
Univ. of P#nnsyiv«nts, 




Mf i H«cht Is ^\cm pr»si<lent 
of m«r»<«tlnir of AOP P»r» 
sonnat 0«t« systsms» inc. 
Ho Is A formar f seuity mw* 
b«r of thft City CoHam '^^ 
Ntw York snd Bt onn Com* 
muntty coiion*, wh«r« rta 
waa alto coordinator of \n* 
atiiutlonal atudloa. A mam^ 
b«r of tha An^iarlcan Pay^ 
chofoftleal Aaa04£*» t^a la co* 
authof of tha booklat 
t«fvl*wln|f Tflchniquaa for 
tha Non^paraofinal CKacu^ 



fchrtmtj 1, 1971 



370 



77 



APPENDIX B 



Paper written by Dr. Samuel H. Cleff describing 
development and initial validation of the Cleff 
Job Hatching System. 



"Project Indianapolis: Patterns of Competence" 
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PROJECT IKDIANAPOLIS 



PATTERNS OF COHPFTENCB 



By: Samuel B. Cleff,i>h. D. 
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In chc spring of 1966, chc auchor was asked Co propose a research ^nd 
developnenc efforc to che Cencer for Ituloporulenc Accion and che N\icional 
AsiiQclnf ion of Hinuf,tcUircrs for a job*person macchlng sysccn). As wich a 
parallel efforc, che syscem was Co be capable of corpucerizacion; buc should 
repre;>enc some obviously differenc and hopefully innovacive approaches. 

Ac chac cime, che auchor was an employee of General Eleccric Co.'s , 
Missile and Space Division as a sore of psychological*managemenC''Con&ulcanC" 
In capcivity. The reporc which follows describes cht resulcs of che research 
and early developuenc efforc. 

An approach was ouclined and discussed wich represencaclves of che 
C^F.I.A^ and che y.A^M. , who were very enchusinscic abouc ic. A proposal , 
was wriccon and accepced, and che research process was begun. Ac chis wricing 
cwo research and dovelopmenc phases have been^coinpleced; and che auchor is 



now deeply involved in che firsc wide^scale field appllcacions of che syscem 

wich L«o Scace of Kew Jersey, The ;:w*:ch Carolina Manpower Develct^jienc Corpor* 

a'cion and wich several large privace companies. 

The Cwo experimencal phases described here were completely supporced 

private funds as pare of che efforc by che Cencer for Independenc Accion co 

decionscrace chac ic is noc only possible buc necessary for che "privace 

sector" CO develop creacive alcematives co governmenc accion on social 

problccis* For chcir faich in che pocencials of the "way ouc" approach described 

hcrt^ cis demonscrated by boch cheir financial concribucions and their eirocional 

supporc, che auchor is indcbced co che following organizacions; 

♦ 

Lilly Poundaclon 

Scern Fnmily Fund 

KaLionnl AS;iOciacion of Kfrs. , 

Indianapolis Chamber o£ Con^mcrce 

3/3 



t\4 



t 



ERIC 



Tho rcporcing o£ che process ^nd findings o£ ,ctio inlclal cvo K & i> p.^ii^ii 

* 

presents a problem. ^'Frojeec Indianapolis'* xas been a demonstracion q£ sotne 
new choorecleal and mechodologlcal appr(>aehes for aeademie psyeholog/^ as 
■well as the developnenc of some new cools for us^ In ntanagemenc^ voeacional 

' guldanee^ and ; job plaecmenc* The report ean be wrlccen from any at chese 
polncs of view^ buc beeause our hlghesc priority euneern has been with its 
apjiiegblllcy Co whac is now ealled **Urban Problc;r*s" {wc usmi Co eall it 
Vaee problems^) we have eXeeCed. Co vrice che reporc for chose idosc likely 
CD use che syscem* 'A separace piece will be wrlccen for cho psychological 
cocxmnicy^ which is most likely co scudy it^ experiment with it, criticize 
It^ and eventually do research which will Improve ou what is described here* 

|or tnxs^are certain, ootn tne r ^earcn approacn and the system It generated 
for iiaacdlace application can be improved upon, and will bo. 

I ' ' ^" ^* Cleff Ph.D. 

. i . -Vice President ' ■ 



• Adaptive Systeiss, mc. ; 





PROJECT INDIANAPOLIS " Dcvcloptnf> n Joh-ro'-Pcrson Matchlnf> Model Research 
and develoriment objectives : To investigate an analytical approach and 
develop the basic methodology for the'Wtchins" of people in the semi/low* 
skilled work force with potentially available job/training situations. To ^{fcj 
validate the concepts and methods developed* and to begin the development 
of a ncchanizable operation iiK)dcl for matching people and jobs at the 
ficmi^skill and low-skill levels. . ^ 

Research rationale end background : 

In considering the approach we wanted to take, we began by reviewing 
the general practices in the field of job-matching; both from the point 
of view of the worker looking for a job and of the cr*ployer seeking 
workers. We found the area cluttered with many ideas and practices 
which date back t.7 ancient Zgypt and the Roman Empire; and most severely 
influenced by the labor-job market of the Great Depression of the 1930*s 




whai jobs were relatively scarce and labor telatively plentiful. Since 
Vorli War II there has been a re"Vrsal of this labor-to-jobs relationship, 
with goverrjoent policies heavily committed to maintaining a relatively 
"tight" labor market. With many jobs going begging, however, there sttll 
remains a sort of irreducible minimum of "hard-core'* unemployed; J" 
minority group members are represented in that minimum at about twice 
I their proportions in the general population. We also found that en^ployers 
\ were seeking people with "skills" needed to do^ jobs which demanded little 
skill, largely because they did not want to make the small investment in 
training. Yet Lnbor turnover in this group is extremely high and expensive. 

From a scientific point of view^ we , found few realistic attec^pts to 
build organizing concepts which would treat the entire field in a 
systematic way and which would generate systematic and experimentally 
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« ccscablc solutions to the problems of human resources management * for 
the benefit of management as well as of Individual workers* The entire 
field is so disorganized and primitive that we felt we had to start from 
so ,w^ch to develop our research approach and logie* 

We began by accepting a psychological logic which appears to be 
implicit In almost all Job-to-persori matching practices over the centuries^ 
that is; In fieneral people tryo to seek out those situations or activities 
in which they perceive the greatest likelihood for success and try to avoid 
those situations or Jictivitles which they perceive as having the greatest 
potential for failure. Employers try to discover the patterns of success/ 
failure for a 'Job applieant^ using many techniques^ but principally the 
unsystematic biographical interview^ in order to '*mateh*' the experience 
patterns of applicants to the requirements of a Job* Job search and 
e\^luation by workers is even less systematie and eonslsts mostly trial 
and error exploration. ^ 

Though the above logic is Ifmplicit in almost all attempts to m^tch 
Jobs a id people^ there have been notably. few efforts to systematize the 
process* Despite the obvious facts that there are no .^obs with«:ut people^ 
a suitable "language'^ or set of variables based on human activities which is 
comon to both jobs and workers has not been developed* It seemed to us that 
this would be an eminently fruitful task in our own research and development 
efforts* It was encouraging to no^^e. that "weighted application blanks" and 
"autobiographical inventories'* have been consistently useful in predletlng 
job- success in a number of different occupations* 

In less complex times it was possible for a person seeking employment 
to have fairly accurate knowledge about the activities content ^f most 
jobs In his liniaedlate environment* In choosing an occupational line or a 
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Job, he had good evidence co flU]^porc his ehoiee and inereosc chc proDaoillcy 
of his sueecsst The t;aine kind of knowledge was available Co empleyers so 

that chey eould fairly aeeuracely deeide abouc che relevanee of an applieanc^s 

C^'^rienec co av;jilable Jobs» As eompiexicy and speeializacion inereascd^ 

che number of ^^bbs*' also increased* Ve have reached che poinc ac which 

che U *S.E.S* Diccionary of Occupacional Tides liscs some A2,500 jobs* Ic 

has been csclmaced thac by che year 1980 some 30Z of all che Joqs lisced in 

che D*O.T* will have been developed in che incervening twelve years because 

of new and changing cechnologies^ Ac che same cime^ a similarly significanc 

proporcion of Jobs will disappear* 

Should we develop new kinds of people Co do these new Jobs and declare 

cjjose new employed in dying Jobs obsolescenc as people? The logic of presenc 

cmploymenc pracciges Implies chis« \fe prefer Co declare che logic of presenc 

praccises obsolece and find or Invent a new and viable logic foi designing 



jibs 



and describing suicable workers* In a Very real sense presenc employ* 
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menc praccises logic assumes thac a worker experienced in a parcicular Job 

classlficacion is capable o£ performing In chac Job classification only* 

Vhac shall we do wich all che bank cellers who will be replaced by che 

aucomaced syscemf now being inscalled and cesced? Can we declare all bank 

cellers obsolece and puc chem on welfare? Presenc methods represent a 

scatic logic crying co operace in a dynamic environmenc* Our view is chat 

Job tides arc largely social lal^ ls which cell us very liccle abouc che 

accivicics conccnc of a Job^ buc which do describe Ics scatus and ocher 
* 

social quallcics* Job cities avc ccrcalnly not nearly as uscxul in 
describing conccnc as chey were when che eeonoG^r was far less complex and 
dynanie* 

In order co know whac a person actually did in a previously held 
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job, an employer roust find out what activiti es the worker- performed while 
holding a job * every experienced «nd well<*tralned cniployinent Interviewer 
knows that* 

It Is apparent to us that certainty about the duties or activities 
performed In an occupational classification Increases as a function of the 
aiDOunt of training, education, and experience necessary in that occupation* 
A person who describes hl^nsdlf aa a corporation logal counsol for « large 
corporation has given a great deal of Information about his probable job 
activities* On the other hand» a person v^a describes himself as a 
'*busboy" has told us very little* except of his organizational social, 
atatus» because his specific duties are more highly dependent on the 
pirtlcular restaurants where he Has been working* What does a "fireman** 
do on a diesel locomotive? 

I .We have tentatively decided that there are five relatively homogeneous 
occupational levels, which are discriminated partially on the basis of the 
extent of training necessary to^per.£orm at average standards within them^ 
and partially on the basis of social responsibility. These are; I. low/seal* * 
- sT:lIIed, where on-the-job tralnlnti is less than 120 days; 2. gKiI]^ect| where 
four months to three years of training is required; 3* Professional, where 
thr^e to ten years preparation is necessary; 4* Msnasement, defined by 
organizational responsibiliry for the efforts of others: and 5. Innovative, 
whlcu is defined by causing technical changes in the other fou.. classif icatlons* 
Uc believe that the dimensions for describing patterns within each of these 
five classes will be relatively homogeneous but significantly different 
between classes. Project Indianapolis focused only the first occupational 
level because of the immediate need in our society and because this level 
probably accounts for about 65% * 75% of all the jobs in the economy of the UtS* 
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Therefore, in fomtulating the research and development approach we * 
worked on the basis of the following assumptions: 

1, There is no scientifically established system presently available 
ia the field of matching people and jobs; instead present practises are a 
conslomeration of superstition, tradition, intuition, and splintered 
scientific efforts, 

2. Since jobs require people, it is both feasible and necessary to * 
develop a set of variables, or dimensions, cc tnion to jobs and people which 
can be used to compare the two; as well as to compare people with each 
other and jobs with each other* . * ^ 

3, In general, human beings tend to seek out and continue those 
activities in which they believe they will be more successful and to avoid 

* Ac*' 

those activities in which they believe they will be less successful, that 

* ^^^^^ 

is there is a positive competence motive. 

4. Jobs and people at the^semi^/low^skill level share relatively ' 

.A 

homogeneous dimensions along which their individual competency activities 
patterns may be described reliably* 

5* There are dimensions of social/emotional motivation which develop 
out of the interaction of competence motives and environmental opportunities 
.which account for a significant portion jof variance in the matching process* 



' Resarrh find Devolopment Plan: * • } 

\ As sujbfgested above, we decided that our first step should be the ; 
devel'^pmcnt of a language with which both, jobs and people could be described* ! 
Xn order to do thif we decided to use the methods of systems analysis; to 

I 

study a sample of Jobs and a sample of poeple who might be considered for 
those jobs, and who in turn, might consider those jobs. By collecting an 
exhaustive sample of human activities from both jobs and people, we could 

U <^ , '. , 



begin with cither set of data atvd develop a reliable classification scheme^ 
then test Its validity by seeing If the same classifications could reliably 
account for the kinds of activities In the othar. Once we had done that, 
we could then develop analytical rocasurlng Instruments based on the 
"language*' we had developed. These Instruments could then be applied to' 
samples of workers and their jobs» with certain knoym or measurable 
characteristics^ to test the basic assumption of general congrulty between 
workers and the jobs they keep , as well as certain corollaries to this 
basic proposition. If this basic congruity concept held up» and the 
hypotheses generated by this concept were validated^ then we would be on 
conceptually safe ground for pursuing the development of an operational 
systecms model bas'ed on those concepts. 

V» also decided that prediction In both directions was necessary for 
the building of a successful operational model. That ls» a good "match'* 
between person and job should predict both good employee performance from 
the point of view of management and job satisfaction, from the point of 
view of the worker. Phllosophlcallif» a good "hire" for a job Is a good 
"placment** for the worker. 

On the assumption that so-called **chronlcally unemployed" individuals 
are representative of the semi/low-skiUed employee group from the point 

of view of the dimensions of their activities^ though probably different 
t ' 

\ln ti.c patterns described along these dimensions^ we decided to use as 
our human sample a group of about 150 chronically unemployed people from 
Indianapolis. It Is generally agreed that their chronic unetaployment Is 
a function of social and emotional variables as well as of variables 
related to their competence or sHill potential. Our sample of jobs came 
froa^ the same Indianapolis labor*job market. 
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PROJECT INDIANAPOLIS 
Phase I Basic Research and System Development 
Method 



A* Samples 

!• Persons 

For the purpose of this study, "chronically unemployed person" Is 
defined as one who, thoug|;i physically capable^ ha8 been unemployed at least 
us much as he has been emflloyed over the preceding five-year period; or was 

i 

predominantly unemployed from age twenty*one If the subject was less than 
twcnty*-slx at the time of the study (summer, 1966)* The subject must also 
have been available for work during the period used to determine his 
employment status; thus housewives, ex*>lnmates of Institutions and students 
were conslderal "employed" if they were engaged In those activities and .^ot 
seeking full<*time employment elsewhere* 

So that age would noc__be the predominant factor in determining 
enplc*'^cnt, the sample was limited to persons between age twentv-one and 
,£orty*flve^ inclusive* Further exclusions from the sample were 1) physically 
handicapped persons, 2) so*called "underemployed" persons In Jobs not 
consistent with skills and abilities unless they met the basic criteria, 
3) frequent Job^changers^ unless employment history met basic criteria* 
Though the proportion of whites ^n the chronically unemployed group was 
double that of non*whites^ within the non*whlte population the rate of 
unemployment yas twice as great as in the whlcc pojfkulatlon of Indianapolis* 
Our people sample, therefore. Includes twice as many Negroes as whites* 
(See Table 1 * for some other characteristics of this group.) 
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RACE - SEX 
EDUCATION 


KEGRO 
MALE 


NEGRO 
FEMALU 


WHITE 
MALE 


WillTE 
FEMALE 


%, 
♦ ■ 


4 


1 

. I 


1 


2 




5 - 


8 


4 


6 


11 


2 


9 - 


11 


23 


. 


7 




12 - 


UP 


19 


23 


19 


2 



47 54 39 4 

RACE - SEX 



21 


- 25 


15 


12 




4 


1 




- 30 


9 


10 




5 

— ' — 


1 


31 


- 35 


6 






6 


2 


.36 


- 40 


7 


13 




4 




41 


- 45 


10 


15 

1 




20 








47 


54 


> 


39 


4 



' TABLE I< SO>IE DE<C?GRAPKIC ClWRACliitilSTICS OF THE PEOPLE 
INTERVIEW SAMPLE, PROJEW' INDIANAPOLIS ' 
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II. Job' Sample 



A sample o£ 57 jobs was selected £or "interview** and analysis. The cooperating 

I i 

companies said that these .were «^11 jobs for which they would seriously consider 

hiring chronically unemployed people. 

' \ 

level fiCiDl-to low-skilled jobs* 



In general these are considered entry- 



1. 

2. 

3. 

^. 

5. 

6. 
^ 7. 

8. 

9. 
10. 
11. 
12. 
13. 
lA. 
15. 
16. 
17. 
18. 
19. 
20. 

21. 
22. 
23. 
2A. 
25. 

26. 
27. 
2$. 

29. 
30. 



Pumping Station Helper 6*^/^ 
trim Line Base Tester i' 
Machine Clerk ^ 
Gas Station Manager I 
Radio Assembler ] ^ 

Granulation Operator (^^j 
Keypunch Operator 
Lab Technician 
Produce Manager 
Meat Cutter 
Dairy Department Head 
Clerk^Typist 

Baker's Helper ^ 
Qranulation Opera tor^C^S^*'^^ 
Pumping Station Helperf^I) 
Weight Master ^ 
Core Maker , 

Oil Filler r- 
Chain Assembler 
ri:ictronlc Instrument 

Assembler 
Chemical Operator 
Security Guard 
Switchboard Operator 
Metal Squeezer 
Mechc^nlc's Helper - 

Elec tronlcs 
Contact Pjint Assembler 
File Clerk 
Automatic Popping 
Machine Operator 
Military Pay Clerk 
Mixer 



TOB-INTERVIEM SAMPLE 



31. Appliance Installation Helper 

32. Capsulation Helper 
**33. Punch Press Operator 
**3^. Lab Technician 

35. Application Investigation Clerk 

36. Shipping Clerk and Bottle Filler 

37. Chief Clerk * Check-O^Matic Section 

38. Meter Cleaner 

39. Final Inspector 
, 40. Marker Checker 

41. Ice Cream Route Salesman 

42. Lead Man Set Up Desmonds 
43'. Cable Assembler 

44. Lead Man - Milling Machine 

45. Bank Teller 

46. Manager Trainee • Burger Shop 

47. ReSrlgeratlo.". Repairman 
^48. Punch Press Operator 
**i49* Keypunch Operator 

50. Telephone Operator 

**51.* Filtration Plant Oper^^tor 

**52. Filtration Plant Operator 

**53. Capsule Filler 

54. Pipe Fitter 

55. Lead Man * Diamond Setting 

Operation 

**56. Keypunch Operator 

<**57.* CapBulte Filling Operator 



it^ Duplication 



1« After the usual "participation" confcrcuccs in New York, Philadelphia/ 
and In Indianapolis with local Industrial personnel people^ the first step 
*as to get a local statistician familiar with Indianapolis to lay out the 
port and Job satnples as described* Then a group of soelal workers from 
Planner House were selected and trained to do the people Interviews* (See 
the following page for the Interview outline*) It uas Interesting to note 
how difficult these very conselentious and helpful people found It to / 
learn an Interviewing approach which did uot ask for feelings, attitudes, 
Interests, motives, and family relationships* Ue v^ent through a morning 
of group training and explanation, then an afternoon of role-playing with 
tape-recorders as very usefuly training deviees* A representative of the 

If ■ ■ 

eontractor remained in Indianapolis to integrate and oversee the gathering 
of the interviews* 

2. Inhere were' 150 people- interviews completed and tapedi six of which were ^ 
Impossible to transeribe from the Capes*j The 'outline for ^'people- interview^*' 
i.s on page 11# ' !• V^JJj / 

3* Using the outline shown on page 12, fifty<-seven Jobs were interviewed 
through questioning both a worker and a supervisor for each job* All 
Interviews were taped* One Interview was unacceptable* 
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OUTLINE FOR PEOPLE-INTrRVIEM 
I* Incroducclon 



"I am 



^, from 



and pare of a research team trying Co help people gee inco Jobs chey 
can do best. So I need your help co find out what you do besc^ and 
would wane Co do If you had a chance." 

II. Establish Rapport (5*15 minutes) r 

Uae profesGional freedosi the guideline loay be **** Do you feel 
eomf or cable? * 

in. Q uestion 1 . 

•'Suppose you begin by telling me about all the Jobs you have had. 
Start with the first job you ever had even if it was while you were 
a kid in sehool " and work up to the present time.'* ^ ; _ 

"I am particularly interested in those things you;did better than 
others on eaeh Job. Now, what did you do on your first Job?" 

(For the first three or four minutes try not to / 
interfere with what they say. If they don't give 
you necessary information* go back and repeat * 
"Can you tell me more about what you did on that Job?" 
Asking when and where they worked ^ds strueture 
and reinforces.) 

After they've concluded the Job part, thank them by 
saying. '*Thank you. That covers the work experienee 
very well." 

IV. Question 2. 

**Ho;^ tell me about your edueation. What did you 
do in school? Start with gr^de sehool and work up. 
What did you do better than other things?" 

(Again thank them .when .they've finished.) 

V. Question 3 . 

JWhat do you do in your free time? What do you 
do better than other thin^^^ in yoUr free time?" 

(Here the person can easily fall into feelings " 
turn it off by getting back* to concrete 
question What did you do how do you do 
it?) 





V 
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OUTLIME FOR JOB tHTI^RVXKtf 

I, Introduction 

,ij ^ working on a Chamber of 

Conmerce research project trying co help describe iobs in a 
better way so that people can be placed more successfully. 

II. Question 1 * 

a) To worV.er -- "Tell me whnt it is you do on this 
Job starting the first thing Monday morning. Begin 
with the very first thing you do when you enter the 
plant, 

1 am particularly interested in the activities which 
you do that make you more competent than others," 

b) To foreman — "Think of the most competent man 
you have and tell me what he does on this job 
starting first thing Monday morning. Begin with 
the very first thing he does when l^e enters the 
plant. 

♦'I am particularly interested in the activities 
which he does which make him more competent than 
others." 

t 

III. question 2 . — ^ 

♦•What were the things you ''(he) did in learning how to 
do this job well? Go back as far as you want in the training 
and education necessary." 

XV. (Question 3 . 

"What is it you do (he does) on this Job to* make it 
more enjoyable?" 

(This leads to question A) » 

V. Question A . 

♦'On what kind of things are you (is he) free to use 
your (his) own judgment." 

(If answered in question 3, don't ask A). » 



386 



At All interview tapes wero transcribed and typed. Content analysis* 
seeking 'behavioral units," were then done on all person-interviews. 
Content analyses were done according to the instructions below. 

/ \ 

' ' COHTEHT ANALYSIS INSTRUCTIONS 

1. Isolate behavioral units. 

a. .A set of words wh^ch describe a unique and 

meaninful activity. Not a job name or 
course name (not "dishwasher** or "math.") 

b. A behavior that has some objective. (Not 
accidental behavior such as winking^ 
scratching or kicking knees.) 

II* Must be a complete message. At leasu a verb and 
an object. (Do I have an image o£ vhat this 
1^ activity looks like or does? Have X communicated 

* the message?) 

III. Must be distinct. 

i * IV.' Firsts underline anything that could be a 

H behavioral uniL. Then go back and condense on 

words of the subject and use b^st way to describe 

activity. 

V. Three people should analyze each interview. In case 
o£ differences » two out of three mist agree. Any^ 
thing that cannot be described should be put aside 
for Dr. Cleff. (Don't use unique name for part of 
machine on Job, use generic term to describe the 
part.) 
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..^^ i/cMU7Jiujiaji uiijictf ^naiyzca out were typed on to address 

labeU for manual sorting by the author* The 3500 ualts gathered from ' 
people-lntervlcus ucre Iccpt separated from the 2400 units collected from 
job-'intervicws. 

6. After reviewing the behavioral units^ and after long and involved^ 
sotQetimes heated^ discussions uith two mathematical statisticians^ ue 
concluded that the data to this poxnt could not be subjected feasibly 
to any known statistical treatment. We had wanted to use cither a 
paired comparisons technique or factor analysis for developing the 
pattern dimensions. The first would have demanded the pairing of 
several thousand items^ each with the othcr^ by several subjects; for 
factor analysis the feasibility problem uas similar* The paired 
comparisons uould probably hove been the better treatment^ ^followed by 
cluster analysis of groupings resulting from It^ but because of the 
great time involved and the pressure to complete the study« it was 
decided to go another way* ^, 

7* The author personally sorted «very behavioral unit according to a 
set of general principles, (See sorting principles)* He then defined 
and described each of the categories which was "discovered" by this 
sorting and asked two other people to classify a sample of 25% of the 
' items he had already sorted* Reliability for the three individuals was 
, quit:: good. There was over 80% :.6Jfc^ent among the three sorters for 
litems which were like the author's sorting, and less than 20% agreement 
on c.*icgory for those sorted diffurently "than the original. 
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Princ iples for Sortin g BQliavlorgX Units Catherod from 
Person Interview into Categories and Sub'Catefiories 

a* Begin with a simple set of general categories, 

preferably classical and philosophical in origins. 

\\ • ■ 

b* Sort a large random saTr4>le of behavioral units into 
these mjor philosophical categories. While sortings 
generate hypotheses for sorting into functional * 
eub*categories. 

c. Test these hypotheses within each major category by - 
trying to sort into hypothesized sub -categories. 
Ceneratejnew hypotheses where it seems necessary. 

d. After having sorted all of the first random sample 
within each of major categories and sub'categories, 
write preliminary definitions of the hypothesized 
sub-'categories* 

e* Sort the remainer of the behavioral units into the 
appropriate sub*-categories, keeping aside those that 
do not "fit" in any. 

f» Cenerate new sub-^category hy;>othescs on the ba^ls 
of those kept aside in Step 5» and/or revise defini- 
tions of some where it seem| necessary^ 

g. Rewrite definition^ of all sub^categories to conform 
to the latest uacible hypotheses which fit 987* 
behavioral units collected. 

h. Ask two or three other judges to sort a large random 
sample of unite already sorted and noted, to check 
reliability of definitions and items sorted. 

An introduction to and the definit^ions themselves of the classification 
system appears on pages 16 *l9s 



I 



1 



IWTRODUCTION TO CATEGORY DEFINITIONS 



The classification system of human coping behavior described 
here, with special reference to a sample of chronically unemployed 
people in Indianapolis, Is based on a major categorization of 
man*s activities orientations into (1) Things, (2) Ideas, aud 
(3) People. This Is a classical approach. I'he breakdown within - 
each of these Tnajor classes Into meanlnful sub-categories 
necessarily followed the activities collected from our sample 
population in Indianapolis. 

Within the People category, what emerged was altcost a re- 
statement of the central thesis of things, people, and Ideas 
in terms of the distance. Involvement^ or responsibility 
involved. Within the Thlnf;s category a similar pattern has 
emerged which goes from the very concrete to the higher levels 
of abstraction, with the tool aud machine Intervening between 
the do-er and the thing. Within the Id e as^ category, the 
relative concrete abstract nature of the syraols the do-er Ueals 
with £e^l out in the sorting of the activities. 
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DEFINITIONS OF 16 SUB- CATEGORIES 



Category T 
Tl 



T2 



T3 



T4 



T5 



for acclvlclcs which directly involve things , 
or tools to work on thlnptS . 



Athletic 



Activities wMch make use of the 



large muscles of the body, with high physical 
exertion and relatively large expenditure of 
energy, or use cools which require the same* 
Such as dig ditches, mix concrete, laod trucks, 
play football^ put the shot, -etc* 

Utility - Helping or peripheral activities 
which do not involve either the skillfull 
use of tools or muscles and whose objective 
is to s^^ve the time or energy of someone else* 
Such as delivering papers, running errands, 
handing tools, stocking bins, etc* 

Fine Manual - Activities which involve 
relatively low immediate expenditure of 
energy, but requiring attention to detail 
because of the smallness of the things worked 
on or with. May work vith hands or tools, or 
light machinery. Such things as soldering 
wires, assembling electronic parts, making 
small things, sewing, knitting. 

Gross Manual ^ I ndependent Activities 
which involve low to average amount of 
energy expenditure, but requiring only broad 
attention to the operation because of the 
grossness of the task or the product, Uork 
is done in relative independence from close 
&uq;)ervision. Such activities as laying 
bricks, painting houses, building construe*^ 
tion forms, framing houses* , 

Gross Manua l - Dj3>9n.4.9.9.^ * To engage in 
activities *which Involve low to average 
energy cxp^enditure rate^ broad attention to 
operations, vse machinery or tend machinery 
in assembly line or under systematic close 
supervision by instruction or by foreman; 
i,e*, grind metal parts, operate dishwasher, 
feed conveyer, etc** 






■ -K*. ^ i^^^ ^^i^-^: 



3^1 



V - 



P2' Service - Activity which implies a reidCiively 

intimate or personal relationship with the 
person or persons whose wants or needs are 
being served; i.e., cut hair, tend patients, 
^ bathe children, etc. 
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T6 Order - Activities whose central function 

is maintaining or improving environmental 
order where one Is working or where others 
arc working or things that are used at work. 
Such activities as housekeeping, tidying, 
sweeping, setting tables, washing clothes, 

etc. ' tafSJi 

T7 Correction - Activities with the objective 

of correcting or maintaining quality in the 
work of others or self. Such activities as 
repair paitv, inspect, set up machinery, 
checking. 

T6 Lo comotion * Activities which involve driving 

or riding in mechanical vehicles, or operating 
equipment mounted on them. Such as drive fork-' H^l^* 
lift, drive truck, drive tractor, ride bicycle* 

K 

Category P for activities directly involving people 
(or domesticated animals) 

PI Attendance * Activity, relatively cursory t^ge 

and imperbuiial, which meets the inmediate " 
wants or needs of another person, but which 
does not involve deep social interaction or 
personal bodily contact; i.e., wait on ' 
customers^ wait on tables, take orders for 
goods or sei:vices,' etc* 
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P3 Mani^gement ^ Activity which implies taking | 

responsibility for influencing the present L 
and/or fucure behavior Of other people; i.e., 

teach, supervise^ hire, plan social doings, etc. tt^*^^ 

P4 A rgument - Activity .with the objective of con* • 

vincing another person to act in a way beneficial 

to/or in concert with the do-^er; i.e., sell, buy, * i 

trade, campaign, preach, etc. ) 



oTtcRory I for activities concerned directly with Idew; or 
symbols of Ideas, people, or things 

^ ^ y<^rbal - Activities In which verbally stated 

Ideas are| central, or where problems are^stated 
and solved In verbal tertns; i.e., read iDSttuc** 
tlons, tafce shorthand^ fill out reports, 
mewwizQ £acts, etc. 

I 2 Numerical - Activities In which nuirierlcally stated 

ideas are central, or where problems are stated - 
and solved in numerical terms; I.e., read compass, 

read gaugei measure fIoor» survey land, count, etc. 

I 

\ 

I 3 Clerical i Activities aimed at recording or keeping 

track of events, or completing procedural paper 
work for administrative purposes; i.e., order ^tock, 
process loan, keep records, sort or file. 

I 4 Artistic * Creative - Activities In which the 

Individual must use artistic talents at some level, 
or must be novel or creative In his approach. I.e. » 
\n:lte copyi draw, draft sketches, arrange window 
displays, etc. 

8* After having completed the sorting and classification, and written sub 
category definitions for the behavioral units collected from the people-^ 
Interviews, the author used the^^ definitions as the base for sorting and 
classifying the units gathered In the job -^.Interviews. See page 20 for 
the sorting principles used for Job data and for the definitions of 
four new sub^^categorles found necessary. About 907. of Job units fell into 
the 16 sub^^categories developed for people data. The remaining lOTi fell 
into the added four classes* .Note the highly plant - or company speclfl 
natui*'^ of these sub^categories* * * - • 
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PRINCIPLES FOR SORTING BE11AV lOR AL UNITS GATHERED 
KROM JQIWIKri-RVIKWS 

A. Begin with the set of categories and sub- 
categories as **discovered" Tor the units 
gathered from the porson interviews 

b. Put aside units vhich cannot be sorted 
reliably, into those categories. 

c. Hyp^th9esize and check out new sub*-categories 

necessary. 

d. If no items are found .to fall into a person 
interview sub-category, reject that sub^^category 
for jobs. 

e. Uritc and re-write definitions as needed* 

ft Ask two or three judges to sort a large 
random sample of units already sorted and 
noted, to check reliability of definitions 
Aud sortings. 
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DEFINITIONS OF FOUR ADDItlONAL SUB<- CATEGORIES 
BASED ON^JOB D ATA 

T 9 Adiustment - Activities which help equip- ty^^ 
* ment adapt to a particular or specific set 
of operations ♦ 

T 10 >toteri<ils & Too ls - Activities involved in 

selecting, obtaining, and preparing^materials ! 
And special tools in order to perfopa other * f 

operations. ^ : 



P 5 Coinmunic<ite - Activities who^se principal- 

purpose is the transmission, person-to-person, 

of Job-relevant information to colleagues^ : . 

superiors, or subordinates. |^ 



394 

- 20 - 



15 Technical - Activities based on a background [ 
of technical experience or education in the 1 
, sciences and technologies. I 



9# On the basis of the srst of sixteen sub^catcf^orles conmon to both people 
fl ild lob Interviews > a first approximation of an Instrument called "Interview ' 
Check List * Persons'* was constructed* It was derlgned to measure the 
rt^uuive tendencies of a pjcrson to approach or avoid activities classified 
according to Xhe sixteen sub<*categories# When scored It provided 4 profile 
which represented the pattern of the Indivi'dual's reported activities 
preferences and experiences. 

This Instrument had a multiple forced choice format* in which the 
respondent Is a^ked to indicate his activities experiences and preferences 
for twenty test groups* Each test group consisted of sixteen behavioral 
ur^ts» one from each of the sixteen sub<>^eategories# Preference and experience 
reports were gotten in order to help the subject differentiate between what 
he has and has not done» and what m likes and dislikes* When ::^re clearly 
differentiated these can be mathematically combined with each other to 
iioprove predictive capacity* ;71thin each te$t group a "forced nC^noal 
distribution" was used to provide /differential scoring weights* .It was ^ 
expected that empirical test would determine the weighting system* though 
4n arbitrary system was used at first* 

It was decided that the profle would have a "real" zero point inasmuch 
as items selected as preferred or done would get positive weighting and 
those selected as disliked and J,east. done would get negative weights* The 
preference and experience profiles for sl givX ^ho worked as a receptionist 
clcrJ;-typist is shown in Figure 1* 
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Plmonslons 
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PIgttr«;i. Profiles of indicated Ikes and dislikes, and activities 
Indicated as done an'd not done. Obtained from the eo 
scoring of an* In terview Check Ltat - Pers ons. 
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10* Ve wanted to construct a Job descriptive Instrument .similar In 
structure to the Intervi<^ Check List - Persons^ but ve found this to be 
difficult tecause of what appeared to be the relative lack of generallzablllty 
in the units collected in the Job Incervlews* So we tentatively designed a 
card sore questionnaire which madcf direct use of the dimension definitions* 
Ve were dissatisfied with this Instrument, however^ and decided ourselves 
to intprove it early in Phase II of the project. 

Flfiure 2 , Is a sample Job profile generated by the Job descriptive 
Instruioents developed In Phase II. 
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Dtmenslons 
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T3 — 

T4 — 

T5 
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<J) ■ 


-40 * .20 0 ' +20 440. 



Fijgure^2j 



J* ^« Non-weighted sum of dimension scores* 

Job description profile generated from scores on 
Job Outline Check List* 
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C General Findings: " • , 

Though Phase I of this study had exploratory objectives with an 
applied end in view, there are some interesting data whieh bear on the 
g^ /al theoretical approach as well as on present practises in the 
selection, guidance, and selection areas. 

1# Of the approximately 3500 behavioral units garnered from the 
people-interviews, about 60X fell into the Th in gs category, about 257* in 
the Ideas category, and about 15% in the People classification* 

2# Of the approxljnately 2500 behavioral units from the job*-interviewS| 
about 60X fell into ThinRS , about 35X into Ideas, and the remaining 57*. into 
People * Within the Ideas classification only a very few units (6) fell 
iui^D the Artistie - Creative sub*-category^ 

Z* In sorting the units taker, from the job interviews, it was 

necessary to ereate four. sub-categories whieh had not appeared in the 

analysis of the units from the people-intervi*ews* Specifically these were. 

called Machine Adjustment, Materia ls and Tools , Technical , and CotnmunicatQ * 

* 

Definitions of these sub-categories are included in the preceding descrip- 
tions of the classification system generated for job-interview units* 
Discussion of General Findings ; 

Most psychological testing now done in vocational guidance and 
in industry generally is eoncerned with our Id ea s category, with a fair 
amount concerned with the People elassif ication* Very little testing is 
done in the Things area* With the activities of our sample of people and 
jobs so heavily thing:;-oriented it would appear that testing is being done 
in * the wrong end of the spectrum of activities* This may help to explain 
the very low predictive validities of psychological testing in industry 
generally^ as well as the relatively high validities of such testing for 
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largely clerical and administrative Jobs^ and for situations with a heavy 
loading in ideas and symbols such as aircraft pilot training and college 
performance* 

When we question the specific differences between the two activities 
classification systems developed here» two possible explanations come to 
mind* First, the four extra sub-c;itegories developed for the job- 
descriptive system are highly plant or compa ny* specif ic and represent 
those areas which get the most attentioa from an employer in early training 
of a new employee* Second, it is possible that failure to discuss such 
activities by the chronically unemployed people, we interviewed helps to 
explain why they are chronically unemployed* That is, since they pay less 
attention to these ]cinds of activities as Important, they lose their Jobs* 
Kost probably the truth lies in some interaction between the two possible 
explanations, and suggests the need for special attention to these four 
areas when people liTce our sample are hired *by industry * 
r* Slscussion of Specific Findin;;c : 

Since the construction of the. two classifying systems is largely the 
wor]c of one person, the author, there is certainly an opportunity for bias* 
The author would have been quite disappointed if he had found it impossible 



25 - 

31)9 



1 



to constrttct two descriptive systems wh^ch were not largely congruent* Yet 
.we felt quite comfortable with th? possibilities for experiment2tion and 

* ' ■ . , i- 

levcntual application to the matching of jobs and people wiich are inherent i .* 

I-:' ' 

in ty.i system* There is certainly room for change and improvc-cnt* For U'' 

7; 

example: though we have sketched out a system for the description of the 
activities conent'-orientation of individuals and jobs, we have not studied L*. 
other probably important orientations such as complexity and status* We 
believe that content-orientation is basic, and that complexity^ status^ and 



othQt rectors may be viewed as volghtlnfjs to the con t en t ^or len ta t Ion of the 

♦ 

Individual and to the content-dmands, of the job requironentat 

As for the measuring , instruments^ wc recoijnizcd that. much remained to 



be done before they could'^l^ecome scientifically or operationally useful* 

It Is to that end that Phase II of Project Indianapolis was planned and 

iioplcnicnted. . ^ , ■ • 

F» Suntnarv and Conclusions'^ Phase I: 

I 

Based on several assumptions implicit in the present primitive state 

I 

of the field of section and placement ve developed an experimental approach 
to studying people-^and-^jobs* With the objective of developing a systematic 
language common to both Jobs and workers at the scmi-low-^skilled level we 
interviewed samples of both* From these taped interviews> using che 
^'behavioral unit" for analysis, we selected the data and developed class-* 
ification systems for people and for jobs which were largely congruent* 
Further, we constructed the first approximations of measuring instruments 
based on the behavioral units and ^he classification systems* 

. have demonstrated the feasibility of developing an analytical 
^^language" com^ion to people and jobs which may be used for matching the 
two, using either manual or mechanized methods* 
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PlIASE II ^ Concept VdIldatlon» Instrument development, and the matching 

loodel* i 

Introductio n 

^ Hotf that we had developed an experimental system of dimensions Along 
vtilch both people and jobs' could be described, and 'had taken a first cut 
At eonstructlng Instruments to measure the relative strength o£ these 
dimensions, It was necessary to Id^ out the general approach to take In the 
application of eventually developed tools/ 

First, we rejeeted the traditional approach of the test developers, 
who attempt to build measuring Instruments which will predict judgmental 
or other so-called **hard" criteria of job success* Second, we decided to 
foj^low the logic of our basic concepts to build a declsionnmaking model for 
the job**to*-person screening process based on some operational definitions o£ 
oceupatlonal adjustment. Third, we decided to use an experimental approach 

f ' 

tof test the hypotheses generated by the model, taking the risk that these, 
would be found wanting and therefore force to abandon or seriously ; 
change our entire thesis of person** job matching based on the beaavloral * 

m 

unit as the analytical building block* 

In general, we followed parallel paths of instrument development apd 
test; and hypotheses development and test; expecting that each would feed, 
back to the other as well as save time ajid money* A detailed description 
^ of the methods we used £ollows* * 
KETHOD 

A* DfjQcnslon definition and.dtcm classification 

Using the sub*-category definitions deserlbed in Phase I, we 
asked a group of nine judges to sort some^ 575 behavioral units into the 
defined 16 sub^categories* The author sorted £lrst and used this sorting 
as a base against which the others were compared* The author classified 



967, of the itms exactly as he had fqr Phase I, For about 607, of the Itons^ 
all eight Judges agreed on classification; in another 10% of the itc:ns^ six 
of the Judges agreed; in the remaining 10% agreement varied from zero to 
"61 This provided us with a large pool of items from which we could draw 
for the reconstruction, of two mea^^uring instruments, 
B, Revising and re-constructing the Self* Interview Check List ; 

We changed the name of the person^tneasuring instrument to 
"Self* Interview Check List" because it struck us that was exactly what it 
had become. - For each of the sixteen dimensions we selected twenty 
behavioral units which were classified with either perfect or near*perfect 
reliability by the judges^ and which -appeared to best represent a specific 
sub^category. Units were then assigned at random to twenty check list 
groupi>, one behavioral unit from each sub-category to each grou^. The 
S , v jc,C.L, , then consisted o£ twenty groups of sixteen units each. As before^ 
the applicant or subject would be asked to indicate his preferences by 
picking from each group the two units he liked best^ the three he liked ^ 
next most^ the two he disliked the most^ and the three he disliked next most. 
Going through each check list group once more^ h& indicated what ha has done 
and not done by picking the two itcns most like what he has done the most, 
the three like those he has done the next most, the tow most like what he has 
done the least, and the three repres editing what he has done next least. 
Different wei^ts were assigned to these choices, 
G. Revising and re*constnicting the Job Outline Check List: 

We made a number of basic decisions which changed the instrument 
for describing Job requirements much more than the person*describlng instnaneat 
had been changed. Firsts we decided on a dual format. Since a Job may be 
considered from both a f^enoral and a specific point of view, we deeided to 
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develop eomplancntary Instruments, whleh when averaged would give a job 
profile onieh closer to the "real** Job requirements than either eould be 
alone* Seeond, we deeided that no cue person should ever be solely 
responsible for generating a Job profile; Instead, this woula be done by 
a superv^^sor working with a trained eatalyst, preferably an Industrial 
englne'^r or methods man. 

For the speelfle deserlptlon of the Job we developed a eheek list 
Identieal In form to the S.I.C.L. , but with different instruetlons, 
Speelflcally, it eonslsted of ten slxteen**unlt groups, one unit from eaeh 
of the sixteen dimensions^ The supervisor pleks those two items in eaeh 
test-group whieh are most like what' has to be done in the Job being 
deseribed, the three whieh are like the next most required aetlvlties; then 
the two whleh are like what is least required and would Interfere with Job 
aeeomplishment if done, and finally the three aetlvitles like those of next 
to Icost value for the job* The supervisor is a^ked to let the items in" 
the eneek list represent or **stand^Ior'* other similar aetlvltle;. We felt 
safe In doing this beeause of the sorting reliability Indleated in **C'* 
above* For the J^O.C.L. we used, only job derived aetlvitles. 

For the general deserlption of the Job requirements we developed 
.a eard sort plus an ^'importanee'* rating ^eale* We re^wrote the slicteen 
^dimension definitions for slinpllt:lty and put eaeh definition^ plus some 
^ainple aetlvitles on the baek, on a 3 x 5 e^rd* ■ Instruetlons are to sort 
these eards into three piles^ five eards to deserlbe what is most required 
by the job and five to deserlbe what Is least required or most interfering; 
the remaining si:; to be in ^ *Meutral" pile. Then the five most required 
dimensions of job activity are rated on an Importanee seale^ while the five 
least required are rated on a non-'lmportanee or interferenee, seale. This 
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rating bXcomcs the nuntcrlcal value assigned to the ten selected dimensions. 

This dual measurement system for che Job made It possible to get 
ftn Internal reliability check, as well as to account for bpth specific and 
gener::l expectations In job requirements. A significant but not extremely 
high correlation between che two measures would serve as Justification for 
pooling th ^ results; a very high correlation would Indicate unexpectedly 

\ ■ 

high reliability and would permit us to use Just the card sort for describing 
Jobs; a very low or zero order correlation would tell us to reject what we 
had done and start all oveif again. 
D. Research design^mechods, and. findings: 

1. We have already pointed out that we intended to test the instruments 
and our concepts and hypotheses In parallel in order to save time and money 
and because of the possible value to be gained from interactive feedback 
between the two processes. And since we were taking an hypotheses-testing 
rather than a crlterlon^predictlon approach w^ took every opportunity to 
load the dice asa Ins t ourselves because of the immediately practical steps 
we wanted to take. This will be discussed more fully in our su;n::iary of 
resultr. Our basic design therefore was to make predictions about relation- 
ships in our measures » take the measures » test tha predictions on a 
probability model, and draw conclusions. : 

2. Our basic organizing principle asserts that, in general people 
seek cut those activities In which they are more likely to be successful 
and avoid those in which they are less likely to achieve this success.. 
Dest>lte the fact that this principle is implicit in present Job-oeeking 
and selecting practises, it needs to be tested^ at least in our terms. Also 
implicit in this process is the Idea that occupatlonally well*adjusted people 
like what they do and believe they are doing welli and further that occupation* 

I 
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«lly well-adjusted people are more likely to stay on a job they have * 
selected than arc those who are occupatlonally maladjusted. In fact, 
present practise labels the occupatlonally maladjusted "chronically 
unemployed," "job-hopper", "unemployable'* ete*^ 

Wo set out, therefore, to test the following hypotheses; 

!• In successfully employed people ; 

. 

A. There Is a positive correlation between workers' 
self-reported activities history and the manage- 
ment-descrlbed requirements of their present 
jobs. 

b* there Is a positive correlation between * - 

activities history and preferences ]. 

c* there Is a positive correlation between preferences 
and job requlrcaients* 

d. workers who report relatively low success In their ! 
present jobs have lower mtches (as In a., b. and 

c.) than those who report average or higher job 
success 

e. workers who report^low success as well as having 
low experience? to-*pre£erence correlations will 
have lower person-to**Job matches than the average. 

f. workers with average or above job success reports 
ajjd average or above preference^-tO'-experlence 
correlations will have above average person<>^to*job 
matches. 

II. Where no criteria for successful employment are used 
to differentiate between work groups except tliat they 
are presently employed, then or zero^order person* 
J tO'job matches will predict work groups with high 

^ turnover and other management signs of Instability 

\ as differentiated from work group's with average or 

\ higher person^to'job matches* 

* 3. Samples , , * ■ 

To test hypotheses a. to f. above, as well as to test the reliability, 

dimension Independence, and field applicability of our Instruments we choso 

a sample of seven companies In Indianapolis. To cross**test hypotheses a* to 

f. as well as the Instability and turnover hypotheses we worked with, the 
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HoncyiJcll £*D*P* Division In Brlghcon, Massachuscccs* ' ^ 

The seven ccmp^nlcs represented a wide range of businesses: Insurance 

company, departrmenc store, telephone companyi bank, and three manufacturing 

f ^erns with a wide range of Industrial products* They provided 23 jobs, 

varying from bank teller^ sales clerk^ electronics assembleri buffer 

machine operator, and phone Installer-repairman to computer operator* Ot}e 

hundred and seventy-seven workers. In these twenty-three jobs, coropletjed 

Self-Intcrvlcw Check List^ * The companies were asked to provide workers 

who had been successfully employed in these j<jbs for a minimum of nine 

months * Of the total number of subjects fewer than 20 had been successfully 

employed less than nine months but more than five months* 

i In keeping with our desire to keep the experiment *'pure'* as possible 

we loaded the dice against ourselves by permitting management to select the 

jpbs they wanted to study, to complete the Job Outline check Lists and 

administer the Self-Interview Check Lists without supervision* Each 

• « 
company was given a research outline and these procedures were reviewed 

wit^ two or three personnel representatives. In this way we tried to come 

as close to actual operating conditions as possible, where rigi'5 experimental 

controls are difficult to enforce* In addition to. providing the check list 

materials, the experimenters also provided a data sheet which included three 

self-evaluation questions concerning perceived success on the present job, 

which when summed provided the me^Fure used in our analyses* ^S^'veral 

companies were asked to administer the S.I.C.L * twice within a period of 

one to four weeks* 

Much the same procedures were followed in the Honeywell part of the 

research, except that the criterion of nine or ^lore months successful 

employment was no£ applied. Instead, llor^eywell management was asked not 

to differentiate in this way, but to pick twelve work groups, some of 

406- 
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which were considered highly unstable. The^o twelve vork^groups htd £rott 

£lve Co thirteen workers each. 

4. Collecting the data and analysis 

' '(Over a period of abouc £our months the companies in question 
administtired the Self^Interview Check Msts and completed the Job Outline 
Check tist_s. As the data came in the Check Msts were scored and every 
reaponso recorded on a fiunsnary data shoot in preparation for later 
transcription to tab cards* All statistical analyses were done by 
computer, though the author did some rough preliminary analsyses by 
scatter diagram and graph. In reporting the results ot these analsyses 
we will begin by discussing the research instruments, going then to the 
testing of the basic concept and the hypo*:heses it generated^ and finally 
to some exploratory work done in model development. 
i It* Self Interview Check List *^ 



* *Internal consistency of the sub-ca>tegories was 
measured in two ways^-by the split-half correlation 
technique and by a randca halves technique* 
Uncorrected correlation^ were very similar for 
both techniques, and were generally quite good. 
Consistencies were higher for preferences than for 
activities history, prcbibly reflecting the very 
low number of scores In certain categories such as 
Verbal and Numerical for our population.. As we 
might expect, consistencies were higher where 
immediate personal experience was available. for 
most of our respondents* ^ 

** Test-retest reliabil ttj; for each of the 16 sub* 
categories varied frotn correlations of +«58 to 
+*93 on preference instnctions, and +.37 to +.85 
£or experience instructions; generally acceptable* 

■-"Te st^retest reliability for the profiles was extra* 
ordinarily high* For experience instructions the 
median interprofile correlation was +.92, with 75*/, 
of the correlations above +.83. For preference 
Instructions the median was +.88, with 757* of tho 
correlations above +*80 . 

^ "Dimension indopendence was clearly demonstrated. 
Mcasurod by intercorrelations aipjng the sub" 
categories and cross-referenced between preference 
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and experience instructions, very few positive 
Intercorrelations were found which were gtatistically 
significant. 

Job Outline Ch<;ck Lis _t 

"" Conceptual consistor icv between the J.O.C.L. and the 
car d sort was on j:he average a correlacion of +.67, 
which though not e>ctreinely high is reasonable for 
this kind of work. We can feel certain that the two 
measures are consistently tapping similar things in 
describing Jobs. And» Just as important, pooling 
the results of the two Job measures is likely to 
gi've us a more precise measure of Job requirements 
than either method alone* 

The basic concept cannot be rejected as a result of our 
data. Table 2. contains the summary data for the 
comparison of personal daOa to Job-description data. 
Using product -moment correlation, five profile compari- 
sons were made for each of the 177 individuals in our 
sample* These were: 1. Preference with Job-Specific^ 
2. Preference with Job-General, 3. Experience with 
Job-Specific, 4. Experience with Job-General, and 
5* Preference plus Experience with Job-Specific plus 
Job*General* In each case -the corrected mean of the 
distribution of interprofile correlations was signi* 
ficantly different from zeio, and positive, as we had 
predicted* Also note that in general the correlations 
between experience and Job profiles ^are higher than 
the comparisons between preference and Job profiles. 
The correlations between experience profiles and job-^ 
specific profiles are the Highest. In fact, the 
correlation of the combined profiles on both sides 
of the interaction are hardly discriminable from the 
experience to job-specific profile correlations* 

Because our predictions are borne out in e^ch instanee 
we can feel much more certain that our bas;tc concept 
of congruity between competency motives ot people and 
the environments they explore i^ a >';enable base from 
which to build our matching model* This is another way 
of saying that the "combined' probabilities" are much 
greater than the probability in any single comparison. 
If one or two of our comparisons had come out in the 
direction opposite to our prediction, we would have had 
to seriously consider some different approaches - both 
conceptually and methodologically* 
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Profile Correlation ' Corrected 
Distributions Mean 

1. Experience + Preference 

vs. ^\ +.42 

Job-Specific 4Job*GeneraI. 

I' 

2t Preference v£.i Experience +.57 

3t Job-Speclflc vs. Job General +#67 

4t Preiference vs, Job-Speclflc +.31 



5* Preference vs. Job-General 



+.26 

6. Experience vs* J'ob-^peciflc +.39 

7. Experience vs. Job^General +*26 



Standard 
Deviation 



.26 



:.32 
.16 
♦ 29 
.27 
.27 
.27 




Table 2 ^ Corrected means and standard deviations for 

distributions of Interproflle correlations as 
Indicated. One hundred seventy-seven incua<* 
bents In twfenty-three jobs for nine months 
or more (about 20 subjects'vere on Job less 
time.) 



^* The hypotheses relating person-to^job match and self* 
reported success was tested by comparing the average 
Interproflle correlations ^of subjects who reported below 
average success vlth those who reported average or betLcr* 
"Success** was measured by iho cotid>inatlon of the ratings 
on three questions: 1, How veil prepared to deal vlth your 
Job are you In comparison to your fellow workers? 2* ^Jhat 
are your chances for promotion, etc*? and 3. How good a 
job do you think you do, etc*? Each Individual rated 
himself on a flve^polnt scale for each of these questions. 
Our **average*' is based on the mean of the response scores^ 
not on the average point In the^^ rating-scale. Total 
possible coitibined score was IS, the mean was 11.3* After 
some graphic analysis we decided that the people who had 
given themselves perfect scores were probably "reaction- 
forming" so we grouped them with the below average success 
subjects for purposes of analysis. Table 3 . provides the 
sucmary data for these two groups. 





I. 




- 35 - 



Profile Correlation 
Distributions 



I. Average and High II. Belov Average * 
, Self-Evaluations Self- Eva luationa 

Mean S. Dj_ Mean S.D. _ 



1^ 

* * 


VS. ! ^ 

.Tnh*5\ni)f^ f f f *4* T/\K**P av^ ^ 1 




, ..26 


+.33 


.32 


* * 


\ 

Preference vs.EKDerience 

\ 


+.64 


29 


'+.ilfi 


.37 


3. 


Job-Specific vs> Job-General 


+.67 


.19 


+.67 


.20 


A. 


Preference V£. Job-Specjific 


+.37 


.26. 


+.24 


:35 


5. 


Preference vs. Job^Ceneral 


+.28 


.26 


+.24 


.31 


6. 


Experience vs. Job*Specific 


+.A1 


.27 


+.34 


.26 


7. 


Experience vs. Jo'b^Ceneral 


+.32 


.23 


+.29 


.26 



Table 3« Corrected means and standard deviations for distributions of 
interprofile correlations of data from two groups of subjects. 
One hundred five subjects with success self-evaluations 11 to 14 ^ 
seventy-two subjects with self-evaluations 6 to 10^ and 15. 
Mean self^evaluation is 11.3 . 

) 

The data confirms our hypotheses th«t those people who 
believe they are less^successful than the average 
employee are more likely ^.to be poorer matches to their 
jobsj in terms of our techniques for describing jobs 
and their incumbents. 

c# The hypotheses relating low sucess report combined with 
low experience*'tO'>preference correlation to the degree 
of job-person match was tested In the following manner. 
First, we defined a group we called "occupational ly 
maladjusced" as being those subjects in our sample who 
hnu below average evalxutioi^ d£ success as well as 
below average preference-to-experience correlations. 
This was our way of operationally defining those peoplu 
who have a history of doing what they don't like and 
feel unsuccessful in their present jobs. We predicted i 
that about IS*/, of our group would meet these criteria 
in a statistical concept of maladjustment. Actually, 
26 or 16*/. of our 177 subject sample met these criteria. 
This group of occupationally maladjusted individuals 
should have either zero order person-to-job matches or 
matches which are significantly ' less than for the 
"adjusted" group. The "adjusted" group had average or 
higher self-evaluations and average or higher preference- 
to-experience correlations. Table 4. presents the 
summary data for both groups. ) 
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Procilc Correlation "Occup-aionally "Occupac tonally 

Distributions ^ Adjustod" I^laladlusted" 







Hcan 


S.O. 


Mean 




1. 


ExDcriiincc -r Preference 

• * 

Job-Specific *h Job*General 




.27 






2. 


Job-Specific vs^, Job-Gcncral 


+.63 


.16 


+.67 


*■ .21 


3. 


Preference v^. Experience 


+.77 


.17 


+.04 


.27 


4. 


Preference vs, Job" Specific 


+.41 


.26 


+.03 


.37- 


5. 


Preference vs. Job-General 


+.31 


.27 


+.05 


.28 


6. 


Experience vs. Job^Specif ic 


+.44 


.28 


• +.29 ■ 


■ :26 


7. 


Experience vs. Job '"General 


+.31 


.26 • 


• +.20 


.22 



Table 4. Corrected means and standard deviations for distributions 
of Intcrproflle correlations from two groups of subjects* 
Twenty-eight subjects VOccupationally Maladusted" (self- 
evaluation below average, pref erence*-to-experience 
correlation below average). Seventy*-nine subjects 
"Occupationally Adjusted" (self-evaluation average and 
- higher, preference^to^experience correlation average 
and higher*) ^ 

1 Our findings here are more startling than any up co 

this point and the table deserves some interpretations* 
First, all differences in level of n^tch between person 
and job are in the predicted direction r that Is, in 
each case the ^^ladjusted" have lower level matches than 
do the "adjusted." Further, some of the -.differences are 
quite large and are statistically significant in ^md of 
themselves. The combined probabilities make it extremely, 
unlikely that these are chance events so we can accept 
them with little fear of being wrong* Beyond the 
obvious conclusions which back up our original theses, 
there are some interesting .data about the process of 
, matching as it now goes on* Notice the largest differ* 
ence we find it fpr preference to job-specific correlation 
* f or the maladjusted the mean correlation is*03, while for 
the adjusted group this mean Is *4l. ' 
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Profile Correlation 
Distributions 

EKpericnce + Preference ) 
vs. ) 
1# .*ob-Spec. + Job-General ) 

2. Jfob-Specific vs V Job -General 

3^ Preference vs* Experience 

4* Prelorence vs* Job*Specif ic 

5* Preference vs_* Job*General 

6. Experience vs. Job-Specific 

7, Experience vs. Job-General 



Eight Stnble 
Wo rk-Groups _ 
Means 

+.49 



+.77 
+.A4 
+.37 
+.31 
+.46 
+.40 



Four low-Stobility 
_^ _ Work-Groups 
M e ans 

+.04 

+.71 
+.21 

-.21 
+.26 
+.21 



Table 5 . Corrected means of distributions of interprofile correlations 
for four groups of workers with low job stability, and eight 
groups with '*adequate^' or better stability. 



T 



f. 



The Itonoyvell Experiment took place in the manufacturing 
plant of Honeywell Inc. EDP Division in Brighton, 
Massachusetts. Eighty-three employees working in twelve 
different factory jobs were administered the Scilf- Inter- 
view Check List , while^their supervisors completed the 
Job ^ulzline Check List with the help of a personnel man 
who had been trained by thtz experimenters. Hanagement 
considered four of the work groups quite unstable and 
considered the other eight satisfactory. A sumstary of 
findings for the^e groups appears in Table 5 . 

The four "unstable" work groups had high employee 
turnover, soice of it deliberately caused by management. 
For instance, one group was a kind of replacement depot 
for workers coming out of one grade and preparing to go 
into another. We were all fascinaaed by the results 
which so clearly different^:;te between the two stability 
groups. Notice how little-account is . taken of the 
preferences of the workeri Yet where the preference*to* 
Job requirements correlation is higher, the turnover is 
lower. These results are xurtarkably like the Indianapolis 
results, if we can compare the more stable work groups to 
the Indianapolis criterion of nine months or more on the 
jo^). Further, the "occupationally maladjusted*' workers 
in the stable groups had lower job-to^person matches than 
did the "adjusted" workers in these groups. 

The Honeywell study confirms the results of the Indianapolis 
experiment and serves as a cross-validation of the basic 
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concepts and the hypotheses generated by them* In * 

addition, it confirms our prediction that cniployee 

turnover is at least partially a function of the 

extent of matchibetween c worker's background and 

the requirement^ of the job into which he goes* It 

Also confirms our hunch that Job success must be 

considered as much from the employees' point of view 

as from management's. Employee turnover is always 

^0 so^Q dcRree a measure of the extent ^ Job 

satisfies an employee* The very low pref erence-co-^ 

job correlations among those in the low-stability 

Jobs strongly suggests this, and predicts expensive 

turnover for the management* 

Discussion of Results t ] 

It should be helpful at this point to review our basic objectives ^and 

to indicate the relevance of the work and findings of ?hase II to these 

objectives* Our prime objective was the development of a system for matching 

people and jobs at the semi-^low^skill levels which can be computerized where 

large numbers of jobs and people are involved* As a step In that direction 

we outlined a matching model whJLch asserts that the optimal '*match" between 

a person and a job is likely to, -be achieved when t^he person's acuivities 

preferences and experiences are highly correlated with the activities 

.dcmar.;^^ of the job. Both job and person activities are to be described ii^ 

the same terms* That soundi like good common sense, but up to now the 

scientific operations leading to a systematic matching of people and jobs 

individually or o_fi tnass o have generally been quite elusive* Therefore, it 

— -I _ ji -I ^ ^ 

was nttessary to test both the oper::tions and the hypotheses gertr;?ted by 
basic concepts in order to come to an operational system for the matching 
of people and jobs in which we can have reasonable confidence* 

From an operational point of view it was necessary to develop 
instruments for measuring and analyzing tHe preferences and experiences 
of people as well as for measuring the activities demands of jobs* To 
these ends, the Self-Tnterviewing Check List was designed to generate a 
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when combined, provided a resultant profile of a person*s relative tendencies 
to approctch and avoid the kinds' of activities represented by the dimensions 
of the profile* T he Job Outline Check List and the Job Card Sort , taken 
tl^cther, providivl a resultant profile of the relative positives-negative 
demand of each Cov the classes of activities represented by the sixteen 
diJBensions of thct ^trofile. 

A basic requirement of any measuring Instrument is its reliability, its 
ability to provitto the same score each time that it is used under similar 
circumstances t in addition, a profile Instrument must dejnonstrate that each 
of the dimensions o£ the profile are relatively independent of the others, 
therefore measuriiu* different aspects of the person or situation, and are 
internally coDsisu*nt« Our data indicate a very high degree of profile 
reliability* Thi:i reliability is so unusually high for work of this kind 



th^t we must ascribe it to the fact that ever^ behavioral unit in the v 
instrument eajne from a group of peqple typically employed in semi*/low* : 
skilled jobst Thorofor^i, little error was added because of the unfamiliarlty 
With Che content* Further, an acceptably high degree of positive independence 
between dimension^^ was achieved; probably due to the care with which units 
were sorted and a*'i::isned to sub-categories* And sitiee internal consistency 
measures were gent:^rnlly satisfactory we can feel quite confident about the 
structure of the j t-l E*interview fcheck*List * 

Since we did no "test'retest" experiment with the job descriptive 
instruments we must have recourse to its analogy to the Self*In*'orview Check 
L^st to support tho need for profile reliability. However, wo did demonstrate 
dimension independence for the Job Outline Check List even bettfer than that of 
the S.I.CiL i as wiill as a high degree of consistency between the two job 
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descrJL;-clvQ instrumcncs* In order to have the samQ high degree of confldenc 

* 

in these Instruments as vlch'the $^I*C*I>* it will be necessary to conduct 

a test**retest experimentJ The Information presently available, however, Is 

adequate to support the continued use of our job^descrlptlve techniques* 

Kavlng a structurally sound set of measuring Instruments Is not enough 

to justify their use In an operational situation* It is also necessary to 

demonstrate that obtained measures are consistent with the set ot basic 

I 

concepts with which we are working, as well as with relevant aspects of the 

1 

**real world'* outside our t;heoretlcal structure* This "real world'' can be 

represented both by other successful theories and by Independent data about 

the people and situations with which we are concerned* Psychologists 

usually call these processes '"construct" and "criterion" validation * 

another way of saying chat the Instruments probably measure what they are 

supposed to measure* 
* 

The presentation of the results data Indicates chat In every respect 

attainted our operations have demonstrated adequate construct validity and 

criterion validity* Vet some discussion of rationale is probably necessary* 

The only characteristic common to che 177 workers In 23 jobs ac ^even 

* 

companies Is that they are considered "successful" employees by chelr 
managcmenc * In addition, about 907. had been on their jobs for nine months 
or longer* The criterion of success Is here considered both from the point 
of Vlc» of the worker and of the management* That is, the tnanag^ment chose 
to keep the worker and che worker chose to scay on che job* Job longevity 
has' mCcinlng only if considered In terms of the Interaction between the 
worker and his job situation* Since we also asked che workers to race the 
degree of success chey believed chey were having In chelr jobs, we can 
conscrue this as a rating of che ntanagemenc's ability to provide a satls*^ 
factory experience for the worker* Note in Table 3 *, that where worker and 
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oanascmcnt success ratings agree the "inateh" between per.-;on profile «nd 
Job profile Is significantly higher than wl^cre success ratings disagree 
(Col. II). This explanation Is reinforced by the data from the Honeywell 
expcrlDient where turnover and Job Instability on a greup basis Is clearly 
a function of "match" between person and job profiles. And In both 
experiments it is clear that the relatively lower degree of attention paid 
to preferences of workers in placing them in jobs (both by worktsis and 
managenient) accounts for much of the low success data. Discussions with 
personnel workers In both experiments reinforced the notioa that relatively 
little attention Is paid to the preferences of seml-/low-6kllled applicants 
when they arc being considered for available jobs. 

If one traces the changes In the "preferences" to "experiences" 
correlations through the Tables, one is struck by the fact that these changes |Hjy 
can be predicted by a theory of personal adjustment developed by Carl Rogers. [Htf 
This theory holds that in general the well-adjusted person describes his 
present self and his Ideal self as being very much the same. Clinical 
experiments with people Involved in psychotherapy have generally borne out 
the theoretical predictions. If we think of preference profile as represent- j^.. 
Ing the "Ideal" and experiences profile as the "re*il" we can make analogous 
predictions for job adjustment. In Tabic 3. the **Averase and High Self* [ 
Evaluations" subjects have higher levels of exp<irience- co-preference ^^i* 
correi^itlons. In Table 5 ., for the Honeywell experiment, the ''Eight stable 
work groups'' have higher levels of experience- to-preferenee correlations 
than do the "Four low-stablllty work groups*" These are precisely the 
predictions which Rogers' clinical theory would have to make. Since | 
management generally pays little or no attention to the preferences of jgg?^ 
applicants for seinl-/low-sill jobs they are likely.to be inducing occupational 
paladlustnetit! The Implications of this for the large group of chronically 
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uncmpl6$cd» or "hard*corc»'* among our minority groups are stiggering- 

• 

Our attention therefore turns to Table 4 ., which, in our estimation, 

t , 

contains the most critical data in the experfjitents. Our operational 
definition of "occupationally maladjusted" corresponds to a clinical 

\\ 

definition inasmuch as It demands a history of low preference^to^experiences 

correlation as well as an^ indication of dissatisfaction with one's present 

situation. This makes it more probably that subjects who meet the double 

I 

criterion are ^indeed maladjusted occupationally, and, therefore, most 
likely to express dissatisfaction by leaving their Jobs. The Indianapolis 
experiment was not designed to demonstrate this, but the Honeywell experiment 
clearly shows the instability and turnover as a funetion of our measures of 
occupational maladjustment. Table ^ * shows the functional relationship 
between degree of person**to-Job match and degree of occupational adjustment* 
Note the zero order correlations between preferenees and Job for the 
occupationally maladjusted group * a finding }«hlch further confirms the 
evidence in ^Tabl e _3 . that management may indeed be Inducing maladjustment by 
generally ignoring the activities preferences of applicants for low-level 
Jobs. The evidence very strongly suggests that If a job-seeker does not 
take his own preferenees into account he cannot depend on the employer to do 
so* Rather the employer tends to make his selection chiefly on the basis of 
"experience*" Unfortunately, he thus tends to reinforce the self-punitive 
cycle of the maladjusted and of the en^loyer in terms of generalized 
dissatisfaction, quality problems, and higher turnover. j 

* It is quite clear to us that the matehing model we have oucxlncd is 
supported by the data of our two experiments. It now remains for us to 
describe the model concisely and analytically, to translate this Into 
operational systems which can be used for actually matching people and Jobs, 
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and to Vicld*tcst the operational system. • 

The Operational Hatchin;; Mod elt 

The evidence of the Indianapolis and Honeywell experiments has provided 

strong evidence for the validity of our Instruments and the concepts from 

which they were developed. An operational system demands that all work^s 

■ \ 

and Jobs being handled by our system can be matched according to the 

principles of optimal adjustment or adaptation* In other vords^ a good 

1 

Job<>^person matching system should promote the well*belng of both the worker 
and the employer. The data available from the experijnents provide possible 
bases for optimizing the correlation between the activities profiles of a 
worker and the demands profiles of a Job. 

Of the Indianapolis sample we found 52 workers In 22 of the Jobs who 
had average and higher self-evaluations ^ average or higher preference* to * 
experience correlations, and average or higher person-to*1ob correlations. 
These are the Individuals considered occupatl'onally adjusted who are 
presently employed In Jobs which match their backgrounds* We decided to 
use this group as the base for designing the operational model* 

Ve had to find out, for each of the sixteen dimensions, the optimal 
relationship of preference and experience to Job*general and Job-speclflc* 
How do "well'^adjusted" workers weight tt^elr activities preferences with 
their experiences In accepting Jobs; In Interaction with employers who 
"select" them for Jobs to which they match at average or bettoi levels? 
In parallel, how do supervisors weight their specific Job demands vlth 
their general concepts of Job demands when ^'selecting" an applicant who 
has "selected" their available Job? In ofher words, how can we 
toathcmatlcally describe the optimal relationships between preference and 
experience on the one hand with job-speclflc and job-'peneral on the jther 
in order to promote the adjustment of the Individual and a good Job done 
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£or the employer? We chose to explore this problem by examining each of the 

« 

Bixtcen dimensions separately^ attempting to discover general principles which 
can be applied to overall profiles as a first approxiinatlon of the model* 

For each dimension vc computed eleven separate correlations of the 
preference score to a resultant job score which consisted of job-specif Ic 
score weighted by 1, plus job-general seore^ weighted by from -5 to +6* 
Exactly the same procedure was followed In computing eleven correlations of 
experience score to job score* Through graphic analysis we determined which 
weighting of job-general have the highest correlation In each series of 
eleven correlations* Figure 3. , the graphic representation of this series 
of correlations for the Locomotion dimension^ Is typical of our findings 
for 13 of the 16 dimensions* In general, we found approxliua tely equal 
correlations for preference- to- job and for experience-to-job at each point* 
The correlations usually rise rapidly as job-^general weighting increases 
from -5 to 0, then either level off at about ,+1, or rise or decline very 
slowly as weighting increases to'^. The general principle emerging from 
this analysis tells us to weight pjreference and experience scores by one; 
and trr weight the job-^specific and job*general seores by one* In other 
words, the simple sucaolng of the two raw scores for each set (or their 
arithmetic mean) appears to provide the optimal matches for our "healthy" 
'well-^matched sample* 

\ dxoce preference and ejcpcrienue scores are weighted equally by the 
Self^Tntervieu Check I^^^sf .* this implies that occupationally adjusted 
workers usually give about equal importance to their own activities 
experiences and preferences when seeking and accepting Jobs for themselves* 
The scoring scales for job-specific and job-^general are quite different, 
however* For job-specifie, the JT ob Outline Check List provides scores 
which can range froa -20 to +20^ while the Job Card Sort provides seores 
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^ w'aich can range only from -^5 to +5. This means thit the Telatiye sizt of 
the Job**specif ic score can reach up to four times th6 valu6 of Job^-general 
raw scores. Statistical analysis reveals that this relative value 
es^ /ically varies from zero to four» with the average at abput 2.6. It 
appears to be a happy aceident that the arbitrary weighting schcne 
originally applied turns out to be about optiioal* We have little doubt 
that} as much more data Is collected, variable weights by dimension for. 
preference and experience and for Job**specific and job-general will be 
Indicated. 

It is very interesting to note that two of the profile dimensions did 
not follow the general "law" exemplified by Figure 3 . , and suggest some 
interesting questions about the effects of supervisory expectations in 
interaction with worker background*. Figure A . shows the preference and 

erience correlations with Job score for these tuo dimensions; Order and 
Fine Manual * Note that the preference*to-Job* correlations vary very little 
around zero. The experience^^to^^job correlations are inverse to the general 
law^ with Order more extremely so* In these cases we are faced with the 
Interesting notion that while supervisory demands increase^ wor^^t^r background 
decreases! Considered interactively, this could me$n that workers with the 
isost ' 6rder experiences avoid jobs with high Or d e r demands and that the 
supervisors of these Jobs do net hire^or keep workers whose backgrounds best 
suit them to meet the Order requirements. The same tendency appears in the 
Fine Manual dimension. 

Aside from the minor anomalies discussed in the preceding paragraph* 
there is an amazing level of correlation consistency from djUnension to 
diinension. For example* the average difference between experlence-to^^Job 
correlation with a Job^Ccncrol weighting of 1 and the optimal correlations 
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with other weightings Is about *05; cxcluslng the exceptional Fine Manual 

* 

*nd Order dimensions already diseussed. For the preferenee-to-job 

correlations this .'.vcragc dlffcrcn c Is .02! The highest correlations 

appear In the Cross^Hanual - Independent and Depen dent, Correetlon , 

LoeoJOtlon ^ Service ^ and Verbal dijuenslons, all +#40 or higher* The lowest 

appear in Fine Manual , Order > Hanagcment > and Artlstle^Creatlve dlmenslorw, 

til zero order or negative* Sueh a distribution might have been ^eeted 

I 

from a sample of workers and ^obs sueh as ours, and the low correlations 

are probably also a function of the relatively low number of choices In 

some of these areas* Certainly,* the seml-to*lou<*sklllcd people and jobs 

examined here are likely to have little experience or Job requirements In 

the Management and Ar tistlc^Crcatlve dimensions* Perhaps the negative 

relationships In the Fine Manual and Order dimensions are also a function 

* 

of our samples* Only further research with other large groups of workers 
and Jobs will clear up this question* The eyldence Is not yet sufficient 
to give differential \jelghts to th« Various dimensions beyond that provided 
by the person and Job profiles* Siinilar analyses o£ the Honeywell experi- 
ment:::! data will be done to search for similarities and dlfferCttces* 

In review then, our data appear to recommend that an operational 
matching model give no differential weightings to preference and experience 

when' combining them to form a resultant profile* The same recommendation 

* * 

Is cK^'^e for combining the job^gencral and Job**specif ic profiles* That Is, . 

the raw scores provided by the Self- Interview Check Llst > the Jpb Outline 

Check List , and the Job Card Sort are to be used without modification* The 

model also calls for the use of the products-moment correlation as the basic 

*^ 

method of determining degree of similarity between a combined person<*prof lie 
and a combined job**prof lle« The use of suitable computer software with 




\ ■ Shovm for "unusual" dimensions only. 
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relatively simple programs will make possible the storage of large **bdnks'< 

of person profiles on the one hand and of job profiles on the other; each 

of which can be searched for above average correlations; depending on 

whether a person Is looking for a job or a Job Is looking for a person* 

Relatively slniple hand analytical procedures will make It possible to do 

these searchcj manually where only small numbers of jobs and/or people £te 

involved* Such procedures are presently under development* 

Projected Developments 

In keeping with our basic concepts, we Intend to develop a parallel 

set of techniques for exploring and measuring the social and emotional 

needs of workers and the social and emotional dezDands of Jobs th^ may go 

Into* A system which helps the matching process by optimizing the copgruence 

between tie emotional and social requirements of both Jobs and people, work^ 

Ing In bonjunctlon with the competency oriented system described here should 
* 

significantly Increase the predictive capacity of the present system** 
Further, since human beings seek'^but complexity levels as well as content 
dimensions in their search for satisfying activities we intend to do the 

research which will make It possible to provide a "complexity weighting*' 

* 

for the presently defined profiles* Such a content*complexlty profile 

* 

fihould prove to be a good predixtor of potential for personal development 

within Job families ^nd hierarchies described by our system. 

I , * * , 

\ Ac the present time (January^ i969) a major program has be^n Initiated 
In the State of New Jersey^ under contract with the Department of Labor and 
Industry, to field test the "Cleff Watching System*' In several Employment 
Service offices* It Is expected that eventually the system will become 
Integrated with the Svate*s Unemployment Compensation and Employment Services 
systems* Efforts are also .underway In several coropjinles^ which aro exploring 



,:^J«-lbU appUctlons for their screening, hiring, and training programs 
" with the "hard-corc unemployed" as well as their more typical employees. 
f^Xhe Office of Economic olrtunlty, working with the North Carolina Manpower 
-Development Corporatlon/L a»barked on a series of field tests and further 
developments of our system to apply to the functionally Illiterate. 
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APPENDIX C 



Statement of Work from the contract awarded 
ADP-POS, Inc. for the implffliientation of the 
New Jersey/Cleff Experiment. 



OEO Contract No. BIC-5261. "Assess and Match 
to Jobs Thirty-five Hundred Disadvantaged 
Individuals Using the Cleff Job Hatching 
System." June 28. 1971. 
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' . Contract Wo. BlC-5261 

Page 1 

SCHEDULE 

CIAUSE I. STATEMEOT OF HORK : 
A* Detailed Requirement 

The Contractor shall provide for a twelve (12) month's period all necessary quali- 
fied personnel, facilities, materials and services (including travel and per diem) 
required to assess and match to jobs thirty^five hundred (3,500) disadvantaged 
individuals using the Cleff Job Matching System (CJMS), 

In the conduct of this work, the Contractor shall carry out the following tasks: 

1* Use the Cleff Job Matching System to match and attempt to place 3,300 

unemployed and underemployed poor persons in the llewark, New Jersey labor 
market area* 

2* Provide for a distribution of Black, Anglo, and Spanish speaking persons 
consistent with representation of each group among the unemployed and 
underemployed* 

3* Provide for^a control group of persons (Black, Anglo, and Spanish^speaking) 
x^roportionally representative oi the experimental group* 

A* Provide for referrals for both the experimental group and the control group 
to public and private sector jobs at a ratio of 1:2* 

5* Provide for recruitment, matching and job .profiling through (1) the Business 
and industrial Coordinating Council (!JICC^ of Newark, ^ieit Jersey, (2) Model 
Cities projects and (3) the New Jersey Rehabilitation Commission in groups 
of approximately 1000-*1200 applicants for each of the above three organic 
zation. The final referral and job placem&nt to be the responsibility of 
each of the above organi:f£ations in accordance with their regular operating 
procedures* 

6* Provide for the data necessary for the evaluation of the program and reports 
specified under Deliverable Items and Schedule below* 

7* Conduct five (5) information and training seminars of approximately two (2) 
days duration each at Contractor's or BICC premises* The information 
portion of the seminars shall provide data on progress to date* The training 
portion of the seminars shall provide training for potential users in the 
operation of the CJMS* The first setninar will be held ^t the end of four 
(4) months of operation, the remainder on a bi**monthly basis* The Contractor 
in conjunction with the GEO Project Manager shall notify participant's from 
the following groups: 

(a) Private employers and Associations 

(b) Public Employer (Government) 

(c) Government Manpower Program operators i^e*, WIN, CEP, KAB, etc* 

(d) Independent Sectors Manpower Program Operators i*e* BICC, OIC, BEU, 
Urban League 

(e) Special Emphasis Programs i^e* Health, Vocation Rehabilitation Migrant 
Workers, Welfare Family Assistance 

Travel and per diem expenses will be borne by the participants 
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ContracL Ko. BlC-5261 



SCHEDULE 



CLAUSE I, STATE^fKNT OF WORK (Continued): 
B* Delivoroble Itemfi and Schedule 

Submit the followins reports and data to the Project Manager during the 
period of this contract: 

1. A statement vithin thirty (30) days of the date of award, indieating 
the random selection process to be used vith the experimental and 
control groups* The statistical basis for determining the size of the 
control group and the subgroupings to be used for the purpose of 
comparisons shall be included in the statement* 

2. Progress report, six (6) copies every two (2) months, by the lOtli 
calendar day of the monthsuceeeding the reporting period, detailing 
the work performed hereunder. The first report shaU detail the first 
sixty (60) days of operation indieating the state of applicant reeruit- 
ment and job profile development; the remaining reports shall include 
dfiLa to be utilized in the seminars* 

3* Six (6) eopies of training materials developed for the information and 
training seminars to be eonducted hereunder* This material shall covert 

<1) the overall operation of the CJMS as an integral part of the 
using organization's operation, (2) the recruitment and training of 
staff for the CJMS, (3) the administration of the Self Interview Check 
List for applicants, (A) Job development using the Job Outline Check 
List, (5) interpretation of results of the matching process (6) 
technical systems operations concerning the contptiter and associated 
activities <7) counseling using the CJMS and (8) general administrativo 
requirements* 

A* Final Report, Six (6) copies and one (1) reproducible copy therefor, 
summarizing the Contractor's total effort hereunder and ineluding the 
training package developed during the contract period, upon completion 
of all work heretmder* 

The Final Report shall include the information collected in the per- 
formance of this contract pursuing to BICC Forms "Appendix D" of the 
Contractor's proposal as well as the following operation data^ 



A. 


Effect 


on 


hiring 


6. 


Effect 


on 


retention 


C* 


Effect 


on 


mobility qtc advancement 


D. 


Effect 


on 


quality of job (wages, types) 


E* 


Effect 


on 


job performance 


F* 


Effect 


on 


turnover (individual and job) 


G. 


Effect 


on 


public ami private job 




Effect 


on 


Spfinish speaking. Black, Anglo 
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SCHliDULE 

/lAUSE I. STATKMl-^iT OF WORK (Continued): 

One (1) copy of each report under 1, 2 and 4 above shall be submitted 
concurrently to the OEO Contracting Officer* 



C# Rei^ortiny; Information 

Reports shall include the following information* 
a* Activity Status 

(1) Selection of Applicants for Experimental Group 

(2) Selection of Applicants for Control Group 

(3) HonHDSO draining and Support Services 

(4) Job Profiling 

(5) Job Placement 

(6) Con^uter Operations 

(7) Evaluation 

b. Statistical 

(1) Number in Experimental Group 

(2) Number in Control Group 

(3) Number of Jobs in Job Bank 

(4) Number of People Placed in Jobs 

(5) Number in Training programs Related to Job Placement 

(6) Number referred to other Agencies for Assistance 
O) Retention to Dat« 

<&) Detailed Data 

(a) Sex and Age 

(b) Ethnic Make-up of Group 

(c) Education Level 

(d) Income tevel Prior to Placement 

<e) Employment History - Individual's Turnover Rate 

(f) Previous Contact with other Agencies 

(g) Arrest or Conviction Record , 
<h) Placement Wage Range 

(i) Placement Wage Average 

(j) Occupational Adjustment indexes 

(k) Hatch Inde):es 

(1) Profile Difference Indexes 

(m) Match/Difference Ratios 



APPENDIX D 



Statement of Work contained in OEO RFP PD 72-54 
for the evaluation of the New Jersey/Clef f 
experiment. 



"Evaluation and Analysis of Job-Client (Cleff) 
Matching Project (in New Jersey)," OEO RFP No. 
PD 72-54, 26 May 1972. 
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lu-r wo. i>u n-'A 



CUUSE I. STATi^ HE ^•^'. OF UORit i 
A . Tinck'T'-unii ; 

■ . . . — ^ . . 

Job-M^-jn K;.rrhiii;; not o new conccjiL* The I>i>:flCt:ory of Occu:^ tio»!;il TitlCii 
orijjinally pnhli-j:Scd in 1939 uas the first syr>LC(nntic nrtompt wo jjcL at the 
problem oi iffl^^rcve<i job plAcom'jnt* The IX)T is a cla:tr,tric;itiou fcclKtio vMicli ' 
/Trv^nfi.?!* Oicupatir.ns accordir,;; to ihoir intvri'elatlonships, standard izer> 
job tiCle^s onU ilCfiincs Rencrally what euch joti involves* As a tool foi: 
idi-'ntifyii^^i occupntion;:! areas and the ^eniiral kindc of job titled jnb« 
which iKCwc in each the COT is useful* Hovjevei:, the kin^la of infonmiLlon 
&v:tilaol(T \\\ z\y^ COT appear to be of little value in nLtci^>linj' to mutch 
iPdivfdunl perils'; to indU^idual job at the entry or m l>Tfit;iiloU*, level » 
An cvLm\*l*^ oi tMt< rii;idity i« that identical IX)T codo:i u;;;!>urc^fvtrtc!nticaX 
job ch^rjicterijsticii uhcn in tuct no two joM arc,idcncic/il, ^^I.jJtCuirQy no 
Lv-o ivtdKii^uais arc idtnticalj yet the asf^isntnent ol; .i:ha'tu::n:a»tDOIi'CCfJe to 
.two indiviau^lft assumes Uiis univc^aL.rolct;ionr*hi[)w 

K.ijor wsknc-s:; of other syptefnS is thf:t they require thnt pc"i»le be <!ei;cribc<I 
on the ba^is of job experience* This approach isi highly <;iief;tionftblf» fron 
scvertil stancpolnts when useu for iiatchinf, at the entry and netni-^Ullled Ic'Vel* 

1) If an individual does not have any job exporiencc, no match can be 

2) IL assur,es that an individual that Uee;; have jcli experience i\Ji 
only suitable for a job In the area of his ex;>6ri«nce, 

3) The prfstriry inechanism for determining an individual' K expcirienci*, 
i^niLaiiiHiy &nd likes and dislikes id the subjective judQemeuL of 
a counselor/interviewer* 



U) Job Analysis tonds to be based priniavlly on en^loyer opinion of 
requirements rather than on scientific asseKGnicnt o£ Job eoniiint* 

In ,*in iT.pt to explore potentially more effective 'apr?VtUches to Uii^rhin;; 
diii:idvnnt;ts.^-:d individuils and jobs aj\d to adHrccft the i;bovcr Ktr^te*! prot>ltw*>, 
the Ofiikt' of rrocriiM Develo: ont ,Comrrainlty Dovelopmcut Division fondod a 
project in Jltwcrlc, Ucw Jersey from July 1971 thru Octiiber 197;? ti> fco«t a 
aoD Mat(!:Ii\^ d,%aten baAod on tlie compiitability of the behiiviur^^l content of 
jobs and I'hv: hrfhavioVal preference and experience of iudivldualit* 

The Clefv ,lnb ^^;!tv^Z^>J System (CJIIS) i% an automated job-pcr^on »wirchin;\ 
^lyj^Li'^n whu*h "iT*atkO)Oi" p«iople at the entry or ;;etni^5)tlllcd luvcl with job 
(ruirini; ^iiinatiorrH* The nuethodolot^y n.ies a tiet of beh;iVioral variiihleH 
co;i<ntm Co ^*i»Lh per::o*i:f r^ni Jobi^ The pi.*rion i$ matched to the Job on ihu 
bifiiK 01 r«ehaviorul prcf- iicncd and eKpericnce. 




SCHEDULE 

ClAUftK U Sr/.Tl'f^i:> n ; Ol* V IOKK (Continued); 

Tlu: CJiIl) iir.f'it a ::rlK*r,ir nf iirot i I iti},; UurAc \;oc\\ reintod liclt.tvt.nrft witlch tlo- 
KcrlSc ImmIi ami Joljfi tu l>**<*t*ij!cly tho :!Ju*/* Irrm!;* Tim U/iiilc 

]>ri*tai!:o Is \\u\t littt'rrulat.iotKsti£))8 o£ tho^t* b^tliav jnrt; (or "dioK'nnlnnft") 
predict boi'h y.noi\ cMiployfjc pcrfotitiniice and periioiuil jol) ^jlf,*ifactiniu 

TI^OF^o '*dimc^i5ioni>" atii incorpor:itcd in a mntchinf^ syi^tcm based on two fiolf- 
admlniiilcvou instritfAcn£;ii, Tho SeH^Intcrviw Check Litft (SICL) is um*d 
to j;et inforc^nlion nu the experience Cactor8» prc£erenceif» and mcitivjit innil 
factors which are uiicd to describe an individiMl in terms of vltat ho has 
dune i\i\6 uyfoi6u6 doi/*;^ nnd what he would liho to do» The Kcn)*tnt; of the 
SICL i^roduces a i^iii profile oi^ tho individual in terms oF his positive and 
ncgAtlvc proCi>tcHiCi ft* and experiences and indicates how well or poorly adjusted 
ho is to work in gencrnit 

Thi! Job Outline Chocl^ List (JOCL) docs for tho jcb yhat the SICL does for the 
person* it Ciet;crihu,s t:l;^ Job in tltcse same sixtciin dimetuions producin); a Job 
proniot The coT^pi]f:'jr cinnpletes the matchiU(f of tho individual to various 
Jobs accordin;; to the ctimpatibllity of the pcr^^oii and Job profile* This is 
not A '*go-iio t,o'* rolntioiiehip bu'L indicates tht> degree to which a person 
mstches a job iu a given environment* 

The project opor;>t<d V) ADD/ri>P» Zne* in Vii^w.-^rU^ Krw Jersey iuvolvos the ut^e 
of the Cleff Jou *tat<:h;ii;; Syatcr;, by several p-iirttcipattn;: a(;eneies» (Sec 
AttaclttPCnL "A" lu'A>jf)* K total of 3,500 p.irt>cipf.nts will he procufc«<!d by 
the )\:rcicipatlt^. ^i|.e>Kk€S usinQ the Clefi .InS Ilutchint; Syst«m» Tho sftonciou 
viU process .1 tonirol proup of BOO participnhtit without nse of the systoM* 
The resuUs of vv;VLriil» plocetttont, and n*to>ilion with both groups will bo 
cofrpnred to cle::rrr)ino the qualit<iriVA di£forpncr*s* The e\ita which will be 
providod by the jirojeet optrotor (AOP) Personnel Data Systems, Inc* is 
as followt;; 

A* ActWlrv S t >; r>g 

(1) Selectlor* of Applicants for Exy rlmtntal Group 

(2) Selection of Appllcanis for Control Group 

(3) Kon-0.*7) Training and Support Services 
(A) Job I'l-ofilina 

(5) Job rtjco^acnt 

(6) Coraputi>r Operations - 

(7) Evaluation 

b* Stntjfitin tl 

(1) KumVr in Experimental Group 

(2) Nuifht r in Couirol Croup 

(3) WuiPb»*j ot Jobs in Job Bank 

(4) UumlM'r of Vcople Placiid in Jobs 

ib) Ntunher in Training ProHVams Ret.iti:d to Job Plncomcnt 
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SCHEDUI^G 

CWUSE I. f;i?;fD:iirJr of l^mK (Continued): 

(6) Number rcitcrrf^d to other i^encieo for AoiintAnct 

(7) Retcnliv^n to Date 

(8) Detailed Data 

(a) Sex and Age 

(b) Ethnic MAke*up of Group 

(c) Education Lcivcl 

(d) Income Level Prior to Pl^^comont 

(&) Eaployacnt History Individual's Tvirnovor Ratu 

(f) Previous Contact with other Agencies 

(g) ArrciKt or Conviction Record 

(h) Placement Uage Range ; 

(i) Pldcencut Wage Average 

(J) Occupational Adjustment indexes 

(k) Match Indexes 

(i) Profile Difference Indexes 

(m) Match/Difference Ratios 
The Contractor shall keep confidential in dll Instances any data identifying * 
Individ. Is. 

Follov-u,> dafia will not br provided by AD?/PDS» Inc. but will be included as 
a part ot the work performed under this evaluation contract! 

B. Purpone of the nvftluntion i 

This rvalv]£tion in to determine whether the Cleff Job Matching System is 
Superior to traditional Job placerjnt methods. Criteria of success are 
primarily Job vetcnlion rates* Job satisfaction* Job performance* Other 
criteria will also be considered, such as placement rate, employer satlcfaction 
with the system. 

1. Analytic Quostiong ; 

(a) A group of low-income inner-city rejvidonts placed in Jobs by use 
of the Cleff Job Matching Sy^^tem will he compared to a control 
group placed by traditional placement agency methods to deter- 
mine if there Any systematic difference favoring one mathud 
over the oth^tr on the following measui'es of "success." 

. Percent referred to Jobs 

• Percent of referrals offered Jobs 

. Percent of referrals placed 

. Percent of total processed placed and offered JtVas 
. Job retention rate 

. KovcflK^nt to ia better Job (salary, stt'itus, position, potential 
for protnotion) 
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Vflpo 9 

CUUr>E 1. STAT^ i n:^ir ok V?onK (Continued): 

'* Jot) nOv;incciacnt * iiormotlon, salary incrc;ifi(! 
i CUcnl latlfitucllon wUh job 

* Jul) pcrtomancc 

i Porcont takluft job^ of those wlio were offered jobs 

* Percent *'no chowjs*' havinti accepted a job oftVT* 

(b) DtK:^: tlie use of CJ143 produce any ctianK<fS In the itf^ency^A 
operations? 

(c) In there a relationship bet^rcun ricoros on the suitability indox 
and the occupational adjustment Index and the success criteria 
(experimental Si^oep only)« 

. Referrals offered a- job 

* Placei\KinCJi 
. Job Kiftcntlon 

Supervisor Ratlni* of Job Pcrfoniunce 

* Client Rating of Job Satisfaction 
i Raised^ Promotions^ Change to a better job 
i Likelihood of accepting an offered job 

* LiUelHiDOdof being a **m show" m an offered job* 

(d) Are there ^lla^^thornc Effects" or other interactions between the 
CJMS and traditional afiency operittionH? 

(e) In the CJl\^ more likely to ph^ce ii pcrtion Into u rcuardinft cnvccr^ 
i.<}^t for those who leave their orit;iiial job* the CJMS client 
moco likely to move on to o better job? 

(f) Wh;U do umplDy*;«rs think of the CJMS? 

1 (CO Wlmt: do the placcni:nt ajjenciCfi think of the CJMS? 
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Paye 10 

SCHEDUU 

STAT1>KNT OF WORK (Conttnued): 
Dependent Vartr^hlos : (Criteria, Output ncr4cuj:cn) 

• Referrals *Xo Joha 

^* R(ifcrrals To Oth(ir Agencies or TralnLng 

• Rcfcrriils Ottevftd Jobs 

• Refonrals Ottevad Jobs Ubo Kcctpt Jobs (Plac(inicnts) 

f 

• Job Quality: Wage Level, Potential Cor Promotion, SES Stctu;^ 
Desirability of Work 

• .Client Satisfaction With Job 

• Client's Job Performance 

• Job Retention 

• Raises, Prcmtionv, Cbangoi To Setter Job 
Independent Varirbtftp ; 

. Type of Placeo^cnt System " CJMS or Ti'cdltional 

• . ;HS Scores 

• Occi2p;stional Adjustment Index 

^ • Match Index t 
^. Profile Difference Index 
I Match Difference Ratio 
I* Sex 

• Etlmlctty 

• Education 

t Employment History • 
.« History of Contact With Other Agencies 

• Criminal Record 
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Kt'l* No. l*D V2-5/I 

SCIIKDUU 

CUUSE 1. STAT^:-;'!'-!? OF WOKl! (Continued); 
C , !3ci.;t1t'<l f^o^ Li^fiy'.n^r. : 

The Contractor j^hill provide nil nocefiffjry fncilitiO!: rtiid c|VKtnt'ie<l por!;onitol 
rcrjuirt^c) lo c jmluc t ar* cvalimtion of tlic OW ClctC Job Matcliinj; Nt^; Jcrr-^iy 
e"-:?crimcrtl ♦ the coiicuct oE this evaluation .ihall include;, but not be limited 
to» the tanks dcccribod below: 

1* AnalyTio cimI report ^ll^data collected by the Projcci includinj;: 

Effectb ol th*» CJMS on Hiring I 

« E£f«>cts of the CJliS on Job Quality (Vlagct^, Type of Job Etc*) 

Differential Effects in Private and .Public Job Markets 

« DirTferential Efiecta among ethnic population (Spanich Speaking, 
Black, White) 

« Effect of CJ1*S on RtiferralSi Acceptance, Placement 

p Eff«*ct of CJMS on Referrals to other agcnclGs or to training 

* Cffetct of Incependenr Variables on Dependent Variable^ 

Cciulti-^Veriatie Analysis Where Appropriate) 

Relationship of CJMS Scares to Placement Effects 

2* tK*tL»rt;;i»u* the current employnont stntua (as oC Nov* 1| 1972) 

for tho 800 mcmbcrfi of the control group iind 3|S00 moiTibcrH of the 
expcrlnontal group* ' 

3* Conduce che foll;>wLng interviews purauant to accessing tho effective 
ness of the CJMS ; 

(a) All mernhers o£ the control n^oup placed In jobs (a:;nume 150 
interviews for budj:etlng purposes)* 

(b) All r.\er.bers of the e>cr*<;rimcni:al group placed in Jobi; (n»60o) 

(c) lirj*:ediato supr^vlsor for original placemc^nt 

* Control Croup (ISO) 

* E:<perlnie.ntal Croup (600) 

(d) 'jO r^jndomly selected personnel offlccra from fUm riin-lf*y'i^f^ 
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5c:iKDiiu: 

CUlISi; I» STATK M^riCr Ol^ WOUIC (ContlmicU); 

QJJ^S clicntn to tliiHerminr wbat they LhlnU of Llic CJMS (Sco 

Analytic quoiicLui (1:) urulor Section R* 1* hereof) I 

(e) :)lri*<:l^«^^; u£ Lhc iil^cctiu^nt a;;c/ Ics involved In the cxpcrlimmt (5) ' 

(f) Two (2) plrtCftinent counsclora from cjich cificncy (10); thcjio 
InLiTvLcws shaXl address analytic qucotLons (d) and (g) undc:: 
Section 1* hereof 

A* l^rovldc Hiisfwors to the anolylic questions net forth unrJcr Section Q* 1* 
hci'cof utlll;{Uitj the results; oi: the Intcrvlcv/s conducted under Pticai;ra[>li 
3 abc/vij and the Collow*-up of Job :;tatus detcvmltiod undci: Paragraph 2* 
above ♦ 

5t Dcter:»uio the rolatloQ&hlp between the Indcrpi^ndent variables and all 
)>05tpl::ct:mont variables (multl"varlate analysis shall be used whore 
appropriate) 

D* Reports : 

1* If thi> Contr;ictor' s proposal Is consldercnj to be an Insufficient 
pvcscntntion of the research de&l(;n» the Contractor i^hall submit 
to the OlO Project tirn ■(;cr six (6) copten of the roijearch dcsicn 
(incliHiinf, a dstallcd pre^eniaLion cf data anulysit; scheme) as 
first tnttiriin report due July 31» 1972* Ko further war!; under thLs 
contract kihall be undertaken until the reoet^rch design Is approvt<l 
by the OEO Project Manager* 

KGTE: For proposal budgeting purposesi ascume this report will be prepared* 

2* l?roi;ress Report, monthly in quadrupllcritc by the l5th day oC each 

c;4lenc^i:r nonth, to the OL'O Project Hcviageri detailing the work perforn:ed 
during the reporting period* ^ 

3* lnt;;riti^ Ac]>ort, ten (K>) copies, due Pecembcr 31, 1972 covering analyni> 
of placc.iM<nt activities shall be :$ubniltted to the OLO Project Manager* 
This report slvould be of such a nature tUcit It can be incorporated 
vcrbwtuir. into the final report* Tour ^^) coiilcs of a^ draft of thi^i 
rcporc shall be submitted by Oc^tober 15» 197'/. Cor review by the OFX> 
l^roject KaVt*i;;er* 

A» FtUil r*;i;c^rt » Due May 30, 1973, 75 copies (lneJudin(; one (I) nproduclblr 
t'.nti;^ri:ring the Contractor':^ total effort hereunder and including Ittt 
rccvMi'*tU;ndations and conclutfions, shall be ouhn.itted to the OKO Project 
N^nagcr* SiN (6) copies of a draft of thLs report shall bu submitted 
by tiarch 15| 1973 for review by the ODD Project M:]im)r,er to the Coniracth: 
Oiflcirr* * ^ 
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ATTACWtCNT 'A 



Hi'V Mo. i'I> 77-j 



CRITERION ' 
MEASURE 



_ ©- 



T 



THIE (APPKOZ. 12 MO.) 



"Before" 
Controls 
N«200 



First 3ra 
Ebci^ariniental 
H-1200 



"BctwOQn* 
Controls 
H»400 



Second 3> i 
ExpQriRi(?nt&l 
H-1200 



"After" 
controls 
ll>200 



Third 
K«12o| 



Description of e>:psriR9ntal prir'-idigm to be followed 
in collection and analysis of criterion d;:ta for individuals 
processed by CJMS (Exp. Grp.) and not pTOccscod by CJKS (Control 
Grp). Grar: linss picture cenerali«ed prcd .Actions made for 
differences bot-rfo-. i eztparir.srvtal and control groups. 



Expcriraantal Groapr ^?iero vill b<? three c;tperiinentf.l groups. 
El, the firct crroup vill consist of cvo-v agency applicnnt 
mcofcing both og-rxy criteria and CCViS ci "toria for placomint 
consideration until 1/3 of the pctrticular agency quol:±\ hcs bue.i 
Taot. E2, tho second group will consist of applicants in 
the second l/3» following the "Betvoen" control period. Sj, 
tho third grerp will consist of rtl.;^ applicants in tJio thir<i 
1/3 following the "After" control period. 

OhesQ oxperiwental subjects are all those indiviOuala proc^.*!iRud 
through the cJJiS by each participating agoncy and are referred 
either to jobs or to formal training on tho banis of their 
ncitch to job or .training situation. An onolic^tnt viiU b o 
rofo rpttf or. lv v f nho-.^'fl .-« .?'.:itnb.iM<'y Trtl tiK of !:0 rr^h i-rhfrr 
to th.it nivMP./' r:r:, .-.v : ' n "v:c.:'"t LolyT ' MV-y.-'- r.^n'. .tVc! :-y' 
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Control Groupp.: Tiicrc v/il3. be throe Oiutinct tiino-clcijcndont 
control groups. C^, the i^OO individuals in "Before" usa of 
CJ^lS. C-s, all 'tf»0 iiidividuala between Experience Group A 
and Evtp^ricnco B procosced v;hcn no one in a particular agency 
i's bsiiVj proccnsod by CJMS (Systcta will be ahat off) and C^, 
the fir^it 200 individuals processed without CJMS imnodiately 
following the completion of Experimafital Group E2* 

r 

Tlio cor^trol subjects must obviously, ac a group, have the same 
deiwgraphic chiractoristics as the ei:periinc:ihal nubjocta and 
nuact the same agency criteria for acceptance as tho exporlniental 
Gubjcct:^ and who have been processed and referred/placed without 
benefit of CJI4S* 

Kota; It dcicldi'd to have twice the nuniber oC 

controls in Coi;trol Group ^^'2 than In #i and tt3 because of the 
critlenl nature of ther. mcaoo-'os and beeauf:o of different 
absolute tiiae:: at which thi:i will occur in the different 
ccopcrs^ting Sigcr^crles aiid locations. ^ 



A seccT^d .type of control group will h'> as well. * Wc will 

cnll thij! an oiporijrental * oelf * conti*ol group. Thi will 
confiiiGt of a >3!nall sample of approximately 50 ap;jlicant;; who 
are indocil piijccd hy uce of tiic COWS. The bac/;ground hirtorie; 
of thc:-:c innividoalc will be studied in depth in order to 
compnro their li::e situations up to the tine that tl;ey vjcro 
placed to what; hapixsns after thoy are placed. 
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3.2 Megymrfin Provxtlca CJHS 

1) Suitability oi; Prcfoircncc to Kxpci-ienco 

2) Difference Ir.dcx of Profei-cnco to Exporienco 

3) suit.*.bility Index of cornbincd Person Drofilo 
to Job Profilo 

4) Diff.-^rcnce Indsx of corcblnGd Person Profile 
to Jcj Profilo ■ 

1) % prccossces referred to jobs 

2) 5i procassce* referred who are offered jobs 
for thone placed 

3) % prccoscoos placed 

4) % referrals pluced 

5) t>l*5cod who ctay on job mor ^ than X*2*3 

6) *ii piacad vmo arc improved in money, status, position 

7) i-'ub^cct and supervisory judgment of job 
p£rfoi".-,«nce 

C) Kovc;r.*.er.i: fro:n job in which placod into another 
job vith another company Where nalari' is higher 
or jTosiponsibility greater or higher status 



ERIC 



440 



APPENDIX E 



Instructions for administering the Self-Interview 
Checklist (SICL) and the Job Outline Checklist 
(JOCL) prepared by ADP-PDS {November 1972). 



o 411 
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AO^ - ^EftSONNei OATA SYSTEMS, rNC. 



INSTRUCTIONS 
for 

ADMINISTERING 
the 
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INSTRUCTIONS FOR ADMINISTERING THE SICL 



INTRODUCTION 



The "sets of client data** regfulred in the Cleff Job Matching 
System to match a specific job or job family (cluster) 
are related to 

1. the clients preferences (likes and dislikes) 

2. the clients experiences (done and not done). 

When the System is to be used by the counselor as a diagnostic 
tool to aid in determining the vocation best suited for the 
client, e.g. as in the Vocational Rehabilitation process 
where most of the clients have physical or emotional dis- ^ 
abilities, a third "set of client data** is regfuired. This 
data is related to 

3. the clients capabilities (cam do and camnot do). 

The first two sets of data - preference • and experience- 
sure collected through the use of the Self interview Check 
List. (See attachments la, lb, Ic, Id). During the collection 
of these two sets of data it is important that the questions 
be answered based on the client's actual preferences and 
(past) experiences; no weight or consideration is to be 
given at this time to the client's current ability to actually 
perform the activities listed. 



METHODS USED TO COMPLETE THE SELF INTERVICT7 CHECK LIST (SiCL ) 

There are two ways of collecting experience and preference 
data using the siCL. The first, and most desirable method, 
(staundard) , is to have the clients describe their preferences 
and experiences by completing the SICL themselves. 

The second method, which is known as the "Direct Rating ■ 
Format (DnF) of the siCL" should only be used in those 
cases where it is not possible- to use the first method - 
e.g., the client is illiterate, the client cannot read 
at the "6th grade level" or above, etc. 
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If the **reading level'* of the client is unknown it is first 
necessary to determine if he or she will comprehend the words 
and phrases used in the SICL* This may be accomplished by 
having the client read the ''reading paragraph" (see attachment 
2a) aloud to the counselor; if the client does not make mere 
than four (4^ mistakes the SICL may be (self) administered 
through use the standard method* If the client makes 
five (5) or more mistakes with the reading paragraph the 
Direct Rating Format of the SICL should be used^ 



IV SICL ADMINISTRATION - STANDARD METHOD 



1. Give the client the blue Self Interview Check List (likes 
and dislikes) and a felt tip pen or a very soft pencil 
(#1). 

2* Tell the client - "This is not^a test - it is a means by 
which you will describe activities that you like and dislike 

- and activities that you have done and not done. There 
are no right or wrong answers and there is no time limit* 
Work at your own pace*** 

3* Tell the client - **You will use this booklet to describe 
activities which you like and which you dislike*? 

4* Ask the client to write his or her name and the date on 
the cover and to turn to Group 1* 

5* Tell the client * "I want you to read the activities 

listed on the page yourself as I read them aloud*" Read 
the 16 activities listed on the page to the client, 
* 

6* Tell the client - "Now I want you to go back through the 
16 activities and this time pick out the two which you like 
the very most and circle the word most alongside of the 
activity and cross out the activity like this^" ^ (show the 
client how by demonstration or by referring to a sample) * 
Wait for the client to make the selection* 

7* Tell the client - "Now go back through the 14 remaining 
activities and pick out the two which you like the very 
least (or even hate) and circle the word least alongside 
the activity and cross out th#i activity like this" - 
(demonstrate)* Wait for the client to make the selection* 
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8. Tell the client * Now go back through the 12 remaining 
activities and pick out the three which you like more and 
circle the word more alongside of those activities and cross 
them out - (demonstrate) . Wait for the client to make the 
selection. 

9. Tell the client * "Now go back through the 9 remaining 
activities and pick out the three which you like less 
than the rest and circle the "word less on the emswer sheet 
and cross them out" (demonstrate). Wait for the client to 
make the selection. 

10. Tell the client - *'You should have two words in Colxunn One 
circled, two words in Column Two circled, three words in 
Column Three circled and three words in Column Pour circled, 
c ' like this.** (Show the client a sample). Now tell the 

[ * client - "Please go back and check you work and then turn 
the page to Group 2.'' 

t 11. After telling the client to turn the page to group 2 and 
I- repeating the process - Steps 5 through 9 above if you 

feel that the client can continue on alone, allow this 

but emphasize the following: 

a. Read all 16 activities listed on each page 
p BEFORE circling the selections. 

b* 1* First select the two he or she likes the 
very most 

I ; 2. Then select the two he or she likes the^ 

^ * very least (or hates) 

3* then select the three he or she likes the 

next most (or more) 
4. then select the three he or she likes the 
next least (or less) 
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c- He or she should check the work before turning the 
page to thie next group to make sure the selection 
is 2"2"3-3. 

i 

I- d. Tell the client to tear out the instruction page 

reminder and put it alongside of the booklet and work 
\ at his or her own pace* 

12* When the client has completed the blue SXCL/ take it (to 
I prevent copying) and give the client the green SICL. 

13. Mvise the client that "you have just described the activities 
which you like and dislike. Now I want you to describe the 
{ ' activities " or activities similar - to those listed in the 

^-^ booklet * which you have done the very most of and the very 

least of (or those you have never done at all) during the 
P course of your lifetime up to now. This applies not only 

ij to jobs you might have had, but also in school, play, at 

hortie as a matter of fact, at any time.** 
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14* Ask the client to turn to Group 1 and you repeat the 
process - items 4 through 9 above for the first two 
groups, 'referring to activities done and not done instead 
of things he or she likes or dislikes* 

15. While the client is working on the green SICL, you should 
spot check the work in the blue SICL - any errors detected 
should be referred to the client for correction* 

16* After the client has con^leted the green SICL you should 
spot check the work and any .errors detected should be 
referred to the client for correction. 



V SICL ADMINISTRATION - DIRECT RATING FOBMAT 

The SICL is administered "orally" by the counselor to the 
j"^ client* This method requires that every activity (16) 

L in every group^ (15) in the SICL be rated first on the basis 

of the client's preference and then on the basis of the 
p client's experience, in other words, you must obtain the 

I client's response to every activity listed in the SICL for 

both preferences and experiences* 

rThe responses for preferences and experiences will be 
recorded as follows: 



1) PREFERENCES (LIKES AND DISLIKES) 

Use the blue answer sheet to record the responses 
beised upon the client*s preferences* The response 
to each activity will be recorded by you, the 
counselor, eis follows: 

, a) Circle (MOST) if the client would like 
doing the activity a great deal* 

b) Circle (MORE) if the client would like 
doing the activity a little* 

c) Circle (LESS) if the client would some-- 
what dislike the activity* 

d) Circle (LEAST) if the client would 
absolutely hate doing the activity* 
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NOTE: If the client hasn't actually performed some of 
the activities listed, ask the following question - 
**If given the opportunity to perform the activity, how 
much would you like or dislike doing it?" Use the 
response to the question to rate the activity* 



2) EXPERIENCE (DONE AND NOT DONE) 



Use the green answer sheet to record the responses 
based upon the client*s experience* The response 
to each activity will be recorded by you, the 
counselor, as followst 



a) Circle (MOST) if the client has done the 
activity or a similar activity, a great 
deal* 

b) Circle (K0R3) if the client has done the 
activity a little bit* 

c) Circle (LESS) if the client has done the 
activity a very few times, (once or twice), 
but never really expects to do it again* 

d) Circle (LEAST) if the client has never 
done this activity, or anything similar 
to it, at all* 



4-17 

- 5 - 



mam 



MP - rCnSONNCl DATA iVSTtMS, INC. 



IMSTRDCTIOMS 
for 

ADMINISTERING 
THE 

JOB OUTLINE CHECK LIST (JOCL) 



ED NOVEMBER 1972 



448 



INSTRUCTIONS FOR ADMINISTERING THE 
JOB OUTLINE CHECK LIST 
(JOCL) 



INSTRUCTIONS TO THE JOB DEVELOPER 



Although most employers and job supervisors will find the 
form o£ the job Outline Check List (JOCL) unusual, the 
majority learn the technique quickly with only a minimal 
amount o£ guidance* 

In introducing the instrument to the supervisors, several 
iir^ortant points should be stressed* 

1, The JOCL is an attempt to have the supervisors 

describe a particular job in a systematic way* It 
is in direct opposition to the varied, non-uniform 
job descriptions offered by most employers which^ 
in the final analysis, give no real indication of the 
type of activities that will be performed on the job* 

2* The JOCL encourages the supervisors to give a great 
deal more thought to th^ sp^ific behaviors involved 
in the job they are describing than they probably 
have in the pest* When completed correctly, the JOCL 
provides us with two vital pieces of data* -First, 
the JOCL profile will show exactly what types of 
behaviors the applicant must perform in order to do 
the job successfully, as Weill as to what degree 
they must be performed* Secondly, and just as im- 
portant, the JOCL will ^how exactly what types o£ 
behaviors the applicant must not perform in order to 
.succeed on the job* These are activities which will 
interfere with good job performance to varying degrees; 
degrees which are indicated in the profile* 

3* With this important information fed into the computer, 
the preference and experience profiles of job applicants 
can be compared automatically, resulting in a good match 
of person to job* Since both the person and the job 
have been described in exactly the same terms, such a 
match is a strong predictor of success on the job 
resulting in the reduction of turnover and greater 
success for the -job applicant* 
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ADMINISTRATKSI INSTRUCTIONS 



The following directions will serve as a guide to you 
for administering the JOCL* 

1* Give a JOCL booklet and a package of tfoCL cards 

to each supervisor* When possible, it is desirable 

to have two or even three supervisors describe 

each job* Th€^^reen JOCL booklets should be used 

to profile '•white collar* jobs? th^^lue are" used 
for "blue collar'' jobs. 

2* Make sure that each supervisor has a soft pencil* 

3* Ask the superv^isors to open the JOCL booklet to 
the first page and read the introduction. 

4. When they have all read the introduction ask them, 
to begin reading the instructions as you read the^i 
aloud* 

5. While reading the instructions, be sure to substitute 
the actuad job title for the word *'job". For 
example, if you were profiling a secretary ^ 

typist job, you would read Step |3 as follows; 

^'Decide which two activities most resemble 
the behaviors most required by the job 
of a secretary ''typist in order to do it 
- best**" ^ 

6* After you have read the instructions and answered 
any questions, let the supervisors turn to the 
first group and begin working* Errors can be 
corrected by crossing out or erasing* 

7* Check each supervisor's work frequently to make 
sure they are completing the booklet correctly* 

8* When the supervisors have. finished all ten groups, 
have them read the card sorting instructions 
while you read them aloud* Let them sort the 
card deck and enter their choices in the appro- 
priate boxes on the chart provided in the rear of 
the booklet* Make^ sure that only one card is 
assigned to each relative degree* 



* see attachments le and If 
see attachments 4a and 4b 
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That is^ there should be FIVE definitions entered in the 
top chart to represent activities which ^KJST be performed 
in order to do the job successfully. These activities 
must he ranked so that one is checked in column S as 
''outstanding'** one in column 4 as "very great" and so on 
down through column I. The bottom chart must be com- 
pleted in the same manner to reflect activities which 
MUST NOT be performed on the job. 

9. When this has been completed tell the participants to 
open their booklets so that the first five groups are 
visible. Ask them to obtain the algebraic sum of each 
of the sixteen horizontal row|. and write the answer in 
the space attlie right. The|?| equals +2; Q +1; 
151 = -* 2 and [J = *-l. To check that this has been done 
correctly* have them add algebraically the "Group 1 S" 
column at the right. Xts sum should equal 0. 

XO. Have them follow the same procedure for groups 6 - 10. 



11. Now ask them to unfold the flap on the last page» and 
I ' transfer the appropriate dimension scores for groups 

1 5 and 6 10* making certain they enter T*-l in the 
space provided for T*-l, T*-2 in the space for T-2^ etc. 



12. Once again an algebraic sum should be derived and 
written in the next column. 

13. The relative degrees of card scores are entered next^ 
with 4*5 to 4*1 placed alongside the appropriate positive 
definition numbers and to -1 placed alongside the 
appropriate negative definition numbers. Nothing need 
be entered for the remaining six definition numbers. 

14* A final algebraic sum is taken and entered in the 
"TOTAL" column. Ask them to obtain the algebraic sum 
of the "TOTAL" column. Xts sum should equal zero. The 
TOTAL column is the job profile which can be plotted on 
the graph. 

IS. After the supervisors hrve completed the JOCL you should 
spot check -their work» steps 9 through 14 herein^ to 
insure that the JOCL*s have been scored properly* 
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; III RECONCILING DIFB^ERENCES 

Our experience has uncovered the surprising fact that 
; some supervisors are uncertain about the kinds of 

specific behaviors that are required to perform success- 
( fully on the job. The JOCL has been constructed to 

; provide information about inconsistencies in this area* 
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For instance^ the algebraic sum of activity groups 
1-10 might yield a total of +9 for the athletic 
dimension T^l* This indicates a job which involves 
the heavy muscles o£ the body for pulling and pushing* 
The same supervisor may have chosen the T-1 card and 
assigned it a value of -5 (highly interfering)* 
Obviously, this supervisor either did not understand 
the instructions for completing the JOCLs, the 
dimension definition , or is uncertain about the need 
for athletic activities on his particular job* when 
something like this occurs, it is helpful to point 
out the inconsistency and try to resolve it by gently 
probing the supervisor about the job which is being 
described* Find out exactly how much athletic 
ability is required and make an appropriate adjustment 
in the T^-l dimension* 

When two or more supervisors are profiling the same 
job, you may find that each looks at the job from a 
slightly different viewpoint* This occasionally leads 
to discrepancies in the prof'rXes which should be dis^ 
cussed with those involved until and agreemtn can be 
reached* When scores for a particular dimension are 
within three points of each other (in either direction) , 
«a straight average can be taken for the final job 
profile* 
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INTERPRETATION OF RESULTS OF THE MATCHING PROCESS AND 
VOCATIONAL GUIDANCE AND PLACEMENT 
WITH THE CLEFF JOB MATCHING SYSTEM 



INTRODUCTION 

Vocational guidance - helping someone to make a vocational 
decision and prepare himself for that vocation - is a per- 
sonal and critical service. It is obviously most critical 
for those individuals whose areas of vocational choice have 
been limited by some physical, emotional, or cultural in* 
capacity. There is little doubt that successful and satis- 
fying work experiences enhance any person's positive picture 
of himself as an independent, self-supporting and worthwhile 
member of society. 

The vocational counselor's most important characteristic is 
a strong desire to help others, to be both sympathetic to 
and empathetic with the needs of his clients. In order to 
do his job well the ideal counselor also has first hand know- 
ledge of the duties and content of hundreds of different jobs 
in many settings and conditions as well as experience with 
a large universe of clients from many walks of life. ThJTough 
his experience he has learned ** intuitively" to match the 
characteristics of individuals to the demands of jobs and 
to consider the effects of certain job placements on his cli* 
ents. As a vocational counselor he serves as a kind of focal 
point and exchange device between jobs and clients, constant* 
ly trying to optimize matches of persons and jobs. Being 
human, he often suffers the frustration of knowing that - 
despite his best efforts - mismatches will occur. Sometimes 
clients refuse to recognize their own best interests^ and. 
sometimes the jobs are not what they were supposed to be. 
The ideal" counselor knows that coercing a client into a 
job choice is Xeally* a •^^no decision** vhich fir^rOeatly leads.-, 
to failure. 

There are many people with the real desire to help others, 
with the sympathy and empathy that make this help possible. 
However, because of the extremely large amounts of time, ef-^ 
fort, and money required to gain the depth of knowledge need-* 
ed about the world of work there are unfortunately very few 
individuals with the "ideal** knowledge of both jobs and peo- 
ple. In less complex times than our own it was possible for 
the curious or interested adult to know the duties and con-^ 
ditions of most jobs in a given locale. In these technolo* 
gically and sociologically more complex times it is well nigh 



impossible for most of us to know first-hand the actual 
content of nore than a few handfuls of real jobs. The 
U.S.E.S. Dictionary o^ Occupational Titles lists and brief- 
ly defines over 35,000 job titles. And the list changes 
and grows rapidly to reflect the technological changes in 
our society. 

It was to help solve this perplexing problem of complexity 
that the CJMS was developed. The system makes it possible 
to explore, via the computer, the thousands of jobs which 
may be available to a job seeker. Trained "job Catalysts" 
develop job profiles in the CJMS language on the actual job 
site? these 16 point profiles are then "stored" in a Job 
Bank in the computer where they can be searched for the 
best possible matches to job seekers' profiles, which are 
independently developed. The keystone of CJMS is a *" behav- 
ioral language" which describes both person and job in pre- 
cisely the same set of sixteen behavioral categories. In- 
dividuals are described in terms of the kinds of activities 
they approach or avoid, and jobs are described in terms of 
the kinds of behaviors an incumbent should approach and avoid 
in order to meet job requirements. Vte call these sixteen 
kinds of behavior the "Dimensions of Work" • 

Each person's profile is a unique combination of these Dimen- 
sions of Work and represents his individuality. The job 
profiles picture the unique qualities of the many jobs. In 
a sense, the computer performs the laborious clerical and 
searching functions which take up so much of a. counselor *s 
tine, permitting him to exercise those qualities of human 
understanding which no computer could ever perform* The 
computer is a filing cabinet which stores the collected in- 
formation classified in terms of the Dimensions of Work. At 
command, it becomes a *'file clerk" and retrieves information 
for the counselor and client based on their needs and demands, 
as they describe these needs and demands. 

THE LANGUAGE OF CJMS 

Because the sixteen part language is essential to the use of 
the system, both by the computer and the counselor, familiar- 
ity with its terms and their meanings is important. It is a 
simple language which is quickly and easily learned, and ra* 
pidly reinforced by use. The guide which follows should be 
used by the counselor to help him as he works with the pro- 
files of clients and jobs, until it has become so familiar 
that the guide is no longer necessary. 
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The human behaviors of CJIIS are classified first into three 
major categories: 1. Thing -oriented; 2. Peoples-oriented, 
and 3. Information and ideas-oriented. Each of these three 
major categories is further divided into sub-categories a.'; 
defined in the list below. Each sub-category has been assigned 
a number and a name which attempts to connote the general mean- 
ing of that kind of behavior. 

T - Thing Oriented Behaviors 

Job activities which immediately and directly 
involve the worker with t hings such that: 

T-1 Athletic 

" uses the large muscles of the body 
in hard physicaf labor 
pays attention to broad overall results 
only 

T-2 Utility 

- helps others by doing unimportant 
things to save their time and energy 

" acts in response to some immediate 
demand by someone else 

T-3 Pine Manual 

- uses fingers in close coordination 
with the eyes 

- pays attention to details and* small 
things 

T-4 Gross Manual - Independent 

- applies some skill in the use of hands, 
tools, or machinery 

- is relatively free from detailed 
supervision by proceduxe, supervisor 
or machinery 

> 

T"5 Gross Manual Dependent 

- applies little skill in the use of 
hands, tools or machinery 

- is dependent on detailed and close 
regulation by procedure, supervisor, 
machinery or assembly line 

T-6 Order 

- puts things where they belong 

- cleans or tidies things 
lubricates things 
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T*-? Correction 

- corrects the quality in own work 
or in the work o£ others 

*- ensures quality performance of 
machinery and equipment 

T"8 Locomotion 

- drives or operates mobile vehicles 

- operates equipment mounted on 
vehicles 

- physically moves away from work 
space a good deal 

People Oriented Behaviors 

Job activities v;hich iI^l^lediately and directly 
involve the worker with people such that: 

P'-l Attendance 

- provides a service completely 
determined by the employer 

- has a very narrow range of decision 
^ has minimal and superficial contact^ 

never physical 

P-2 Physical Service 

- gives a regularized service which 
meets their personal needs or demands 
is perr.itted a relatively wide range 
of decision within specified duties 

- can have physical contact , but a super-- 
ficial personal relationship 

p*3 Management 

guides^ influences^ or directs the 
present and/or future ongoing beha^ 
vior of others 

takes or shares responsibility for 
results of ^that behavior^ including 
their work ^ 

- there is high potential for emotional^ 
relationships 

P-4 Persuasive 

- convinces and persuades others to 
react quickly in some way beneficial 
to employer 

takes little or no responsibility 
for future behavior of other persons 
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I - Ideas and Information Oriented Behaviors 

Job activities vmich immediately and directly 
involve the worker with ideas and symbols 
such that: 

I-l Verbal 

- uses written or spoken words to 
represent ideas, people or things 

* describes situations and relation* 
ships f or solves problGms, using 
words 

1-2 numerical 

^ uses numbers or number concepts to 
represent ideas # people or things 

- describes situations and relations- 
ships ^ or solves problems # in numer* 
ical terms 

1-3 Clerical 

- records or orders data ot any kind 
^ processes paperwork intended tc do 

that 

1-4 Innovative 

- gives a personal interpretation to 
a situation and finds an artistic^ 
creative or original solution to it 
or change in it 

- expresses some personal idea non** 
conventionally 



Using the Person Profiles 

The administration of the Self Interview Check List provides 
a "Likes and Dislikes'* or Prefej^ence Profile as well as a 
**Done and Not Dpne" or Experience Profile for the usual job 
applicant* For rehabilitation clients # a third person- 
profile, the "Can and Cannot Do*' of Capabilities Profile is 
also developed. The administration of the *'Can and Cannot 
Do** evaluation provides an opportunity for client and coun- 
selor to consider together, and to Qompare, their percep-^ 
tions of the client's disabilities from a job*f unctioning 
point of view. Because this* evaluation is vocationally rele 
vant, rather, than another review of the client* s structural 
difficulties it should help direct his thinking towards 
those kinds of activities he is capable of doing. And since 
this is done in terms of activities related to success in 
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specific vocational areas, it should enhance the client's 
view of his own possibilities- The search of the Job Bank' 
profiles with the three personal profiles will produce 
matches at different levels to specific jobs, taking into 
account his preferences^ experiences, and capabilities - 



Interpreting the Person Profiles 
— Likes and Dislikes (Preference) 

Extreme scores of any kind^ always provide more information 
about a person than do average scores siiaply because they 
indicate how a person differs from most other people. 
These discrininable differences are the traits and charac- 
teristics we usually hold so important to our individuality 
and self concepts- In the Person Profiles any .score which 
falls between +5 and -5 can be considered both a group and 
individual average, or natural characteristic of the 
person* s preference. 

1. Go down the list of positive scores over 5- These 
indicate the kinds of behaviors the client probably likes 
the most- Certainty about behavioral preference increases 
with the. size of a Dimension score. For example^ a score 
of +15 on T-1 - Athletic suggests a high certainty that 
the client does like to engage in large muscle, high energy 
expenditure activities. With a score of +7 there would be 
less certainty. A score of +22 would indicate a very high 
degree of certainty. 

The same generalizations hold true for negative scores. 
These indicate the kinds of behaviors and activities the 
client likes the least. The greater the size of the neg*- 
ative score the more certain we can be that it ropresentJ 
a *'real aversion** for a particular Dimension of activity. 

2. Another kind of extreme to look for is in the total 
pattern of scores, the profile. Some profiles are ex- 
tremely flat with all scores between +5 and *5- A 
person who produces a flat profile is saying in effect, 

**I don't particularly like or dislike any kind of activity, 
and therefore is not motivated in any particular direction . 
The general counseling problem is to help the client form 
some sense of direction for himself and to make a choice 
of job or training. The counselor's continuing approval 
and support for an area the client does choose is usually 
necessary to assure continuity of effort in that direction. 
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On the other hand, some prof^iles are all peaks, both neg- 
ative and positive. This type of person is saying in effect, 
*'I am very sure about what I like and don*t like", and there- 
fore strongly motivated to go in sorae directions and just 
as resistant to going in others- The general counseling 
problem is one of providing rational forethought and re- 
straint i getting the client to *'slow down" long enough to 
take some long terra objectives into account in his vocation- 
al planning* - 

For most clients you can e:cpect 2 to 5 high negative and 
positive peaks, 2 to 5 moderate peaks, the remainder fall- 
ing within the neutral zone* 

3. Apparent conflicts of interest may also show up in a 
preference profile* That is, there may be two or more high 
scores which seem to be mutually exclusive, either, negative 
or positive* Sometimes there is a high positive score which 
appears to contradict a high negative score* For example, 
a profile may indicate a strong preference for T-1 Athletic 
and for 1-3 Clerical. The first is a large-muscle high- 
energy expenditure Dimension. The second is concerned with 
ordering information and ideas and is usually done in 
relatively sedentary low-energy expenditure situations* Be 
cause these tend to be mutually exclusive people usually 
satisfy one drive in work and the other in play if they have 
this kind of interest conflict* Suppose a profile shows a 
high preference for P-*4 Persuasive and dislike for 1*1 
Verbal* How does one persuade others without the use of 
words? Though it can be done/ it is highly unusual* The 
need to explore these kinds of contradictions with the client 
is obvious and can lead both him and the counselor into 
important decision paths* 

It is very important that the conflicts indicated or sug- 
gested be reviewed with the client so he can be helped to 
work out for himself the understanding and re-ordering of 
the preferential priorities necessary for vocational choi3e 
and development* It is often enough to help him recognize 
that the conflicts do exist, to identify .them specifically, 
in order to help him take into account these things in his 
vocational planning* In the exercise below these^ factors 
will be discussed in greater detail and depth* 

4* Preference Profile Interpretation Exercise 

Let us consider the following profile of a young woman: 
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LiKGS Dislikes Profile 



Miss Mary X» 



Tl-Athletic 
T2^Utility 
T3-Fine Manual 
T4-Gross Manual 

Independent 
T5-Gross Manual 

Dependent 
T 6 "Order 



-25 
- 8 
+15 



+ 3 



-18 
-21 

- 5 
+10 
+19 
-10 
-15 

- 7 



T7-Correction 
T8 -Locomotion 
PI- At tendance 



P2'-Physical Service 



P3-Management 
P 4 -Per suasion 



11- Verbal 

12- Numerical 
I 3 "Clerical 

I 4 -Innovative 



+24 
+19 
+11 
+ 7 



A» First- examine the aggregate pattern of her preference 
profile- For thing oriented activities her scores are mostly 
neutral or negative and the same is generally true of her 
People oriented scores* Yet, all of her Information/Ideas 
oriented scores are positive. From an overall point of view^ 
she prefers intellectual activities to either social or con-- 
Crete world activities* Drawing some tentative first hyp- 
otheses about her as a person we might guess that she is a 
thinker rather than a doer, perhaps a little withdrawn from 
the outside world and somewhat "bashful" with people - 
which is often called "introverted"* 

The picture of Miss X takes on some form and color when 
we consider her individual Dimension scores; with strong 
positive interest in Fine Manual, Attendance, Verbal and 
Numerical kinds of activities; moderate interest in Locomo- 
tion and Clerical activities* She shows a strong aversion 
for Athletic, Gross-Manual Dependent, and Order; and a mod- 
erate disinterest in Utility*^ Service and Persuasion, Notice 
that with the exception of a moderate liking for Locomotion, 
all of her positive preferences are for relatively sedentary 
and passively oriented kinds of behavior while her dislikes 
include most of the more active outgoing kinds of activities* 
We might guess that she satisfies her needs for aggression 
* in her driving of cars and in some "creative^activities. 
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C# Fler overall pattern of many positive and negative peaks, 
v;ith only four neutral scores, suggests that Miss X is 
fciirly firm in her likes and dislikes* Though externally 
on the passive side, she is relatively certain that this is 
tha way she wants it to be. ifc would probably be quite dif*- 
ficult to make changes in her self concept or to get her to 
take training or a job which demanded an aggressive outgoing- 
ness such as managing a custcr. dress shop, or even being a 
housewife, which requires Management, Service, and Order kinds 
of activities which might be difficult for her to accept* VJe 
would expect her to defend har relative passivity with some 
stubbornness* 

n* VJhat conflicts are apparent in her Preference Profile? 
In general the pattern is non-contradictory* Even her liking 
for locomotion *- the vehicle moves is essentially sedentary 
in nature* The positive Innovative score suggests that she 
"locomotes" in fantasy or artiscic behavior* It should not 
be difficult to match her into a job which, satisfies both 
her preferences and aversions* 

— Done and Not Done (Experience) 

The Experience Profile indicates the relative degree to .which 
a person has actually engaged (or avoided engaging) in each 
of the Dimensions of Work * according to their owa report 
on the Self Interview Check List* We can never be absolutely 
sure of the exact degree to which these scores represent the 
person's true history* We can be highly certain that it is 
a better account of real activities history than is the Pref- 
erence Profile* The Experience Profile summarises what the 
client says he has done and not done, in terms of the Dimen-^ 
sioifeof Work* It is always useful to review, if possible, 
the responses of a client in discussion with him* 

Just as it is helpful to view the Preference Profile as a 
picture of what the client believes is ideal for himself, we 
may consider the Experience^: Profile as picturing his per- 
ception of vjhat has been real in his activities history* 
Though the same analytical suggestions for interpretation of 
the Preference Profile apply to the Experience Profile, it 
is important to make this discrimination between the *'ideal" 
and the "real''* 

1. Note the extreme Dimension scores. The greater the 
absolute value of a' score, the more likely it is that the 
client has indeed engaged or avoided that kind of activity. 
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2. Look for eKtreir.es in profile pattern. Inffirenco and 
interpretations are analogous to those considered for 
Preference Profiles. The flat profile indiccitos a person 
who has had no definite directions to his life and probably 
needs help in setting direction. The profile v/ith many 
very high positive and negative scores suggests a petrson 
v/ho has aggressively pursued certain directions and just as 
ago-ressivGly avoided' others. Both types of extreme patterns 
usually have difficulty finding vocational opportunities 
which ompletely satisfy them. 

3. Conflicts and contradictions can show up in the Exper- 
ience Profile^ too. Since these represent real-life sit; 
uations they are much more difficult to resolve in counsel- 
ing because they are less susceptible to intellectual and 
rational manipulation and compromise. A person nay drop or 
de-emphasize a wish found inappropriate with guidance and 
help^ but it is impossible for him to "drop" or eradicate 

a real set of experiences, though he may change his percep - 
tions of those experiences ." Contradictions in activities" 
done and not done present a very real challenge to the 
client and counselor. The mutual examination and careful 
review of the profile and its implications for the client 
makes a, helpful basis to counseling. 

4. Experience Profile Interpretation Exorcise 

Let us consider the following Experience Profile of the 
same young woman; 



Done and Not Done Profile 



Miss Mary X 



Tl-Athletic 
T2"Utility 
T3-Pine Manual 
T4-Gross Manual 



-22 
+ 4 

^20 



Independent 
T5-Gross Manual 
Dependent 



- 6 
-12 

- 7 
+ 5 



- 6 



T6-0rder 

T7-Correction 

T8**Loccmotion 



Pi-Attendance +18 
P2-Physical Service -*10 
PS-Management ^ -15 

P4-Persuasion ' + 5 



Il-Verbal 
1 2 ""Numerical 
IS'-Clerical 
14 -Innovative 



+18 
+25 
+22 



4 



4bt 

- 10 - 



A. The aggregate pattern, considering only the major 
categories, is much the same as for the Proferance Profile. 
Her teinclency has been to avoid thing -or ientoci activities 
and people-oriented activities as well} and she had 
engaged largely in Inf ontiation/Ideas-oriented 
behaviors. 

B. With the same kind of extreme scores, the overall 
pattern of variability suggests what we inferred from 
the Preference Profile. There appear to be some strongly 
fixed behavior orientations v/hich are probably not likely 
to change much in which she moves behaviorally . The 
most likely changes are those which would enhance the 
degree of already discriminated directions. 

C. She has no readily apparent contradictions or con- 
flicts in the Experience Profile. Those suggestions of 
possible conflict which appeared in the Preference Profile 
are apparently x'esolved in her Experience Profile. 



Comparing the Preference and Experience Profiles 

The. computer terminal printout provides a summary score 
called "Occupational Adjustment index". This is the 
product — moment correlation of the two proriles, thus 
serves as an index of the relationship be'zween the two. 
The score can vary from a -1.00 to a +1.00. Our research 
has £ound that for a large group of semi/low skill employees 
with demonstrated job stability the average range of this 
index was from +.40 to +.75 while for a group with demon- 
strated instability this index varied on the average from 
".35 to +.35. Other evidence in addition to the foregoing 
has led us to conclude that the Preference to Experience 
correlation is a reliable indicator of "Occupational 
Adjustment" . 

In more common sense terms we can assume that a person 
who habitually does what he does not like/ or avoids 
doing what he likes or further has difficulty adjusting 
his environment to meet some of his important likes and 
dislikes, can be termed "Occupationally Maladjusted''. And 
we assume that the inverse is true of the "Occupationally 
Adjusted". Many factors may lead to or cause occupational 
or behavioral maladjustment - cultural and environmental 
forcrs may have deprived an individual of opportunity or by 
conflicts of personality forces within himself. Obviously 
occupational maladjustment must be explained as the " 
result of a complex interaction of both personal and en- 
vironmental factors. The CJMS has attempted to 



takG these ideas into account in an opa^retional codcl without 
putting the "Blar^ie" either on the envirorcneat oi: the person, 
anci atSciuots to readjust tho relationships b^two.en prefer- 
ence^ e:^perience and job requirements to provide the indi- 
vidual with optimal opportunity for successful interactions 
with his vocational environment. 

In using CJMS, the counselor can observe the following rules 
of thumb about the "Occupational Adjustment Inde:-:*'. Any 
score below 4- J2 5 is a strong indication that the client may 
indeed be occupationally maladjusted and is in need of 
• special attention in the job matching process as well as of 
counseling help. 

A client with a low Preference to Experience index should 
not be referred into a job or type of job unless his Person 
to Job match index is over +.60. This makes it even more 
certain that both his preference and experience has some 
positive relationship to the requirements of the job, thus 
giving hiia a chance to succeed at the job and to recover 
soiue personal stability. A client with a very high Prefer^ 
snce to Experience inde'X can safely be refeirred to a job 
where the Person to Job match index is as low as +.40 
because he will bring personal stability to the job situation* 



Vocational Capabili ty Evaluation 

In a rehabilitation setting, where many or most of the clients 
have some sort of physical or emotional incapacity which can 
interfere with vocational adjustment, it can be very helpful 
to administer the Vocational Capability Evaluation* The 
client and counselor work together to produce a profile which 
describes the client's relative ability to perform tasks of 
the kinds delineated in the Dimensions of Work used for the 
Preference and Experience Profiles. In other words, the 
capacities and incapacities of the client are described in 
vocationally functional terms rather than in structural or 
medically functional terms* Since the '^language" is the sama 
in which jobs in the CJMS are described it .makes it possible 
to match clients and jobs on a functional basis as well as 
on a preference and experience basis* 

The same kinds of inferences and interpretations which 
were drawn from the first two profiles can be drawn here, 
too, though it must be based on the idea of possible present 
functioning, rather than on projected ''ideal" desires or 
"real** activities history. As the Preference Profile pic- 
tures what is desired or wanted and the Experience Profile 
* pictures the behavioral history, the Vocational Capability 
Profile pictures v'hat is relatively possible and impossible 
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for the client to actually perform. The very fact that 
the Vocational Capability Profile is generated by the 
client and the counselor together, plus vjhatever other 
individuals may have an excellent understanding of the 
client^ provides a unique opportunity for client and coun- 
selor to v/ork together in the interests of the Client. The 
objective nature of the instrument provides a mediiim for 
the client to gain a more rational understanding, and accept-^ 
ance of V7hat he is indeed capable of doing and not doing, and 
it provides a platform for counselor ^nd client to stand to- 
gether as a teaxa. 

How the Computer^ Tielpk ^ 

Up^to this point the computer has been used as a high speed 
clerk, recording, sorting, typing, making simple mathemat" 
ical calculations - operations which could be performed by 
the counselor or assistant, but which would consmnc hours 
of their time. The computer operator and her remote slave 
does all this in about six to eight minutes. Once the Self 
Interview Check List is scored and the profiles generated, 
the really helpful work of the computer begins* 

The machine contputes the correlation between Preference 
Profile and Experience Profile, the "Occupational ?.djust- 
ment index" we discussed above. Then it mathematically 
combines the two-profiles into a single profile which rep-^ 
resents an optimaT weighting of the two according to a 
research developed formula. Then it performs the matching 
of this combined profile to all the job profiles in the 
job bank by calculating a product-^moment correlation for 
each as well as a difference measure called the "Difference 
Index"- Ordinarily it would take about three hundred hours 
to perform these operations for a job bank or over 250 job 
profiles by hand machine calculations " the computer does 
it in a few seconds*: 

The correlations, called the "Match Xndex** indicate to 
what degree the shapes of the profiles are alike. The 
"Difference Index" indicates the extent of differences 
between profiles and helps to break ties among match indexes^ 
For example, if a client's profile matches a number of jobs 
at approximately the same level of match index, the one with 
the smallest difference index would be the best match. The 
computer terminal provides a print-out of the ten or 
twenty best job matches in rank order according to the size 
of correlation between client profile and job profile, xf 
a Vocational Capability Evaluation has also been completed ip 
then the computer is required to produce another set of 10 
or 20 matches based on correlation of the Capability Pro- 
file with the Job Profile. Those jobs which produce the 
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hi<jhoi;t prof i le^to-profilc nuitchas on both lisit^ are 
obviously the jobo for v;;iich th:i clic^nt is boat suited ♦ 
Thin printout oZ person to job matches alijo servec as a 
verj' useful counseling tool* 

Both client and counselor should review the raatching print 
outs, the jobs rfipresented, and all of the other factors 
involved with getting and keeping the jobs representing 

thfLT>t 

This listing should provide valuable and concrete inforn'.a^ 
ticn to the client about the kinds of jobs he can expect, 
their pay rates, their locations, etc. Most importaiit it 
supplies a realistic base fro3 which he and the counselor 
can nake their placement efforts* 
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CLE?F J03 MATCrriKG SYSTEM 



R^-ilatiorship o£ Diffctrenco Index to Match luda x 

Tho colunn headed "normal range" iridicateo the expected distri- 
bution of Difference Index at the specified Match Index level* 
If an applicanji's Diiference Index exceeds the range for a 
given Match Index, there is likely to be highly significant 
difference in one or more dimension comparisons, which serves 
as a '*red flag^ to the intervievrer ♦ The lower the Difference 
Index for a given Match Index, the less likely is a major 
dimension difference between the applicant's profile and the 
job profile* 



MATCH imE*/ DIFFERENCE I^fDEX DIFFEREN^CE INDEX 

median normal .range 



90 


to 


94 


280 


200 




3 60 


85 


to 


89 


460 


340 


** 


580 


80 


to 


84 


640 


540 




740 


75 


to 


79 


820 


705 




935 


TO 


to 


74 


980 


850 




1130 


65 


to 


69 


1140 


980 




1300 


60 


to 


64 


1250 


1070 




1430 


55 


to 


59 


1380 


. 1180 




1580 


50 


to 


54 


1450 


1220 




1680 


45 


to. 


49 


1540 


.1280 




1760 


40 


to 


44 


1600 


1340 




1840 


35 


to 


39 


1660 


1370 

• 




1920 


30 


to 


34 


1710 


1410 




1980 


25 


to 


29 


1770 


1440 




2050 
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